2023-03-26 11:53:29,606 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d 2023-03-26 11:53:29,619 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-03-26 11:53:29,634 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-03-26 11:53:29,634 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/cluster_2110af56-5f7e-313d-fc62-08bf66541fb8, deleteOnExit=true 2023-03-26 11:53:29,635 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-03-26 11:53:29,635 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/test.cache.data in system properties and HBase conf 2023-03-26 11:53:29,636 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/hadoop.tmp.dir in system properties and HBase conf 2023-03-26 11:53:29,636 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/hadoop.log.dir in system properties and HBase conf 2023-03-26 11:53:29,637 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/mapreduce.cluster.local.dir in system properties and HBase conf 2023-03-26 11:53:29,637 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-03-26 11:53:29,637 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-03-26 11:53:29,758 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-03-26 11:53:30,192 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-03-26 11:53:30,197 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-03-26 11:53:30,197 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-03-26 11:53:30,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-03-26 11:53:30,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-26 11:53:30,198 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-03-26 11:53:30,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-03-26 11:53:30,199 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-03-26 11:53:30,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-26 11:53:30,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-03-26 11:53:30,200 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/nfs.dump.dir in system properties and HBase conf 2023-03-26 11:53:30,201 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/java.io.tmpdir in system properties and HBase conf 2023-03-26 11:53:30,201 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/dfs.journalnode.edits.dir in system properties and HBase conf 2023-03-26 11:53:30,202 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-03-26 11:53:30,202 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-03-26 11:53:30,822 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-26 11:53:30,827 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-26 11:53:31,111 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-03-26 11:53:31,290 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-03-26 11:53:31,311 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-26 11:53:31,354 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-03-26 11:53:31,386 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/java.io.tmpdir/Jetty_localhost_40411_hdfs____ylmy2q/webapp 2023-03-26 11:53:31,533 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40411 2023-03-26 11:53:31,550 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-03-26 11:53:31,550 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-03-26 11:53:32,140 WARN [Listener at localhost/46069] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-26 11:53:32,217 WARN [Listener at localhost/46069] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-03-26 11:53:32,235 WARN [Listener at localhost/46069] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-03-26 11:53:32,241 INFO [Listener at localhost/46069] log.Slf4jLog(67): jetty-6.1.26 2023-03-26 11:53:32,247 INFO [Listener at localhost/46069] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/java.io.tmpdir/Jetty_localhost_42629_datanode____ef6ply/webapp 2023-03-26 11:53:32,345 INFO [Listener at localhost/46069] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:42629 2023-03-26 11:53:32,751 WARN [Listener at localhost/46409] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-03-26 11:53:33,273 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x493b021343dc5255: Processing first storage report for DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8 from datanode cd68a317-7bd1-46b9-b40b-44ceb88f195a 2023-03-26 11:53:33,274 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x493b021343dc5255: from storage DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8 node DatanodeRegistration(127.0.0.1:37237, datanodeUuid=cd68a317-7bd1-46b9-b40b-44ceb88f195a, infoPort=33693, infoSecurePort=0, ipcPort=46409, storageInfo=lv=-57;cid=testClusterID;nsid=517188794;c=1679831610899), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-03-26 11:53:33,275 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x493b021343dc5255: Processing first storage report for DS-b9c7113b-a3af-49a5-acfa-cf67a55b797f from datanode cd68a317-7bd1-46b9-b40b-44ceb88f195a 2023-03-26 11:53:33,275 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x493b021343dc5255: from storage DS-b9c7113b-a3af-49a5-acfa-cf67a55b797f node DatanodeRegistration(127.0.0.1:37237, datanodeUuid=cd68a317-7bd1-46b9-b40b-44ceb88f195a, infoPort=33693, infoSecurePort=0, ipcPort=46409, storageInfo=lv=-57;cid=testClusterID;nsid=517188794;c=1679831610899), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-03-26 11:53:33,355 DEBUG [Listener at localhost/46409] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d 2023-03-26 11:53:33,417 INFO [Listener at localhost/46409] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/cluster_2110af56-5f7e-313d-fc62-08bf66541fb8/zookeeper_0, clientPort=53955, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/cluster_2110af56-5f7e-313d-fc62-08bf66541fb8/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/cluster_2110af56-5f7e-313d-fc62-08bf66541fb8/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-03-26 11:53:33,430 INFO [Listener at localhost/46409] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=53955 2023-03-26 11:53:33,440 INFO [Listener at localhost/46409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:33,442 INFO [Listener at localhost/46409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:34,131 INFO [Listener at localhost/46409] util.FSUtils(471): Created version file at hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1 with version=8 2023-03-26 11:53:34,131 INFO [Listener at localhost/46409] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/hbase-staging 2023-03-26 11:53:34,465 INFO [Listener at localhost/46409] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-03-26 11:53:34,980 INFO [Listener at localhost/46409] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-03-26 11:53:35,012 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 11:53:35,012 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-26 11:53:35,012 INFO [Listener at localhost/46409] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-26 11:53:35,013 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 11:53:35,013 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-26 11:53:35,158 INFO [Listener at localhost/46409] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-03-26 11:53:35,231 DEBUG [Listener at localhost/46409] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-03-26 11:53:35,325 INFO [Listener at localhost/46409] ipc.NettyRpcServer(120): Bind to /172.31.14.131:43599 2023-03-26 11:53:35,336 INFO [Listener at localhost/46409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:35,338 INFO [Listener at localhost/46409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:35,360 INFO [Listener at localhost/46409] zookeeper.RecoverableZooKeeper(93): Process identifier=master:43599 connecting to ZooKeeper ensemble=127.0.0.1:53955 2023-03-26 11:53:35,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:435990x0, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-26 11:53:35,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:43599-0x10120ac29a00000 connected 2023-03-26 11:53:35,437 DEBUG [Listener at localhost/46409] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-26 11:53:35,438 DEBUG [Listener at localhost/46409] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 11:53:35,442 DEBUG [Listener at localhost/46409] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-26 11:53:35,449 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=43599 2023-03-26 11:53:35,449 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=43599 2023-03-26 11:53:35,450 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=43599 2023-03-26 11:53:35,450 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=43599 2023-03-26 11:53:35,452 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=43599 2023-03-26 11:53:35,459 INFO [Listener at localhost/46409] master.HMaster(444): hbase.rootdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1, hbase.cluster.distributed=false 2023-03-26 11:53:35,531 INFO [Listener at localhost/46409] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-03-26 11:53:35,532 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 11:53:35,532 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-03-26 11:53:35,532 INFO [Listener at localhost/46409] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-03-26 11:53:35,532 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-03-26 11:53:35,532 INFO [Listener at localhost/46409] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-03-26 11:53:35,537 INFO [Listener at localhost/46409] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-03-26 11:53:35,540 INFO [Listener at localhost/46409] ipc.NettyRpcServer(120): Bind to /172.31.14.131:38971 2023-03-26 11:53:35,543 INFO [Listener at localhost/46409] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-03-26 11:53:35,553 DEBUG [Listener at localhost/46409] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-03-26 11:53:35,554 INFO [Listener at localhost/46409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:35,557 INFO [Listener at localhost/46409] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:35,559 INFO [Listener at localhost/46409] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:38971 connecting to ZooKeeper ensemble=127.0.0.1:53955 2023-03-26 11:53:35,562 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:389710x0, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-03-26 11:53:35,563 DEBUG [Listener at localhost/46409] zookeeper.ZKUtil(164): regionserver:389710x0, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-26 11:53:35,565 DEBUG [Listener at localhost/46409] zookeeper.ZKUtil(164): regionserver:389710x0, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 11:53:35,565 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:38971-0x10120ac29a00001 connected 2023-03-26 11:53:35,566 DEBUG [Listener at localhost/46409] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-03-26 11:53:35,567 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=38971 2023-03-26 11:53:35,567 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=38971 2023-03-26 11:53:35,568 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=38971 2023-03-26 11:53:35,568 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=38971 2023-03-26 11:53:35,569 DEBUG [Listener at localhost/46409] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=38971 2023-03-26 11:53:35,572 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:35,581 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-26 11:53:35,582 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:35,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-26 11:53:35,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-03-26 11:53:35,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:35,602 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-26 11:53:35,603 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,43599,1679831614279 from backup master directory 2023-03-26 11:53:35,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-03-26 11:53:35,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:35,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-03-26 11:53:35,608 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-26 11:53:35,608 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:35,611 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-03-26 11:53:35,612 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-03-26 11:53:35,706 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/hbase.id with ID: e8148231-a3be-4dc6-a21c-7281a1b72264 2023-03-26 11:53:35,750 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-03-26 11:53:35,765 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:35,803 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x59d3e3c2 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:35,836 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5302c6e8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:35,860 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:53:35,862 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-03-26 11:53:35,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-03-26 11:53:35,879 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-03-26 11:53:35,881 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:35,885 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:35,887 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-26 11:53:35,922 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store-tmp 2023-03-26 11:53:35,953 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:35,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-26 11:53:35,954 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:53:35,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:53:35,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-26 11:53:35,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:53:35,954 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:53:35,954 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-26 11:53:35,956 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/WALs/jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:35,981 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C43599%2C1679831614279, suffix=, logDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/WALs/jenkins-hbase4.apache.org,43599,1679831614279, archiveDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/oldWALs, maxLogs=10 2023-03-26 11:53:36,051 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37237,DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8,DISK] 2023-03-26 11:53:36,060 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:36,131 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/WALs/jenkins-hbase4.apache.org,43599,1679831614279/jenkins-hbase4.apache.org%2C43599%2C1679831614279.1679831615991 2023-03-26 11:53:36,132 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37237,DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8,DISK]] 2023-03-26 11:53:36,133 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:53:36,133 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:36,137 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-03-26 11:53:36,138 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-03-26 11:53:36,198 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-03-26 11:53:36,206 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-03-26 11:53:36,235 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-03-26 11:53:36,247 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:36,254 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-26 11:53:36,256 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-03-26 11:53:36,274 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-03-26 11:53:36,279 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:53:36,280 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61331402, jitterRate=-0.08609089255332947}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-26 11:53:36,280 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-26 11:53:36,282 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-03-26 11:53:36,308 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-03-26 11:53:36,308 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-03-26 11:53:36,312 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-03-26 11:53:36,314 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-03-26 11:53:36,351 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 37 msec 2023-03-26 11:53:36,352 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-03-26 11:53:36,388 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-03-26 11:53:36,395 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-03-26 11:53:36,430 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-03-26 11:53:36,435 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-03-26 11:53:36,437 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-03-26 11:53:36,443 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-03-26 11:53:36,447 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-03-26 11:53:36,450 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:36,452 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-03-26 11:53:36,453 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-03-26 11:53:36,469 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-03-26 11:53:36,475 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-26 11:53:36,475 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-03-26 11:53:36,475 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:36,475 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,43599,1679831614279, sessionid=0x10120ac29a00000, setting cluster-up flag (Was=false) 2023-03-26 11:53:36,495 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:36,502 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-03-26 11:53:36,504 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:36,509 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:36,516 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-03-26 11:53:36,518 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:36,521 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.hbase-snapshot/.tmp 2023-03-26 11:53:36,573 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(952): ClusterId : e8148231-a3be-4dc6-a21c-7281a1b72264 2023-03-26 11:53:36,578 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-03-26 11:53:36,588 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-03-26 11:53:36,588 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-03-26 11:53:36,592 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-03-26 11:53:36,595 DEBUG [RS:0;jenkins-hbase4:38971] zookeeper.ReadOnlyZKClient(139): Connect 0x23e83673 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:36,614 DEBUG [RS:0;jenkins-hbase4:38971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2d2fdc7e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:36,615 DEBUG [RS:0;jenkins-hbase4:38971] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@29105d03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-26 11:53:36,646 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:38971 2023-03-26 11:53:36,651 INFO [RS:0;jenkins-hbase4:38971] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-03-26 11:53:36,651 INFO [RS:0;jenkins-hbase4:38971] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-03-26 11:53:36,651 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1023): About to register with Master. 2023-03-26 11:53:36,654 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,43599,1679831614279 with isa=jenkins-hbase4.apache.org/172.31.14.131:38971, startcode=1679831615531 2023-03-26 11:53:36,674 DEBUG [RS:0;jenkins-hbase4:38971] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-03-26 11:53:36,681 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-03-26 11:53:36,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 11:53:36,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 11:53:36,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 11:53:36,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-03-26 11:53:36,695 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-03-26 11:53:36,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-26 11:53:36,696 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,699 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1679831646699 2023-03-26 11:53:36,701 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-03-26 11:53:36,710 DEBUG [PEWorker-2] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-03-26 11:53:36,710 INFO [PEWorker-2] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-03-26 11:53:36,713 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-03-26 11:53:36,718 INFO [PEWorker-2] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-26 11:53:36,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-03-26 11:53:36,721 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-03-26 11:53:36,722 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-03-26 11:53:36,722 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-03-26 11:53:36,725 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,728 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-03-26 11:53:36,730 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-03-26 11:53:36,731 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-03-26 11:53:36,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-03-26 11:53:36,734 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-03-26 11:53:36,736 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679831616736,5,FailOnTimeoutGroup] 2023-03-26 11:53:36,737 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48863, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-03-26 11:53:36,739 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679831616736,5,FailOnTimeoutGroup] 2023-03-26 11:53:36,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,740 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-03-26 11:53:36,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,742 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,758 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:36,760 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-26 11:53:36,762 INFO [PEWorker-2] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-03-26 11:53:36,762 INFO [PEWorker-2] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1 2023-03-26 11:53:36,779 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1 2023-03-26 11:53:36,779 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:46069 2023-03-26 11:53:36,779 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-03-26 11:53:36,784 DEBUG [PEWorker-2] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:36,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-26 11:53:36,785 DEBUG [RS:0;jenkins-hbase4:38971] zookeeper.ZKUtil(162): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:36,786 WARN [RS:0;jenkins-hbase4:38971] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-03-26 11:53:36,787 INFO [RS:0;jenkins-hbase4:38971] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-26 11:53:36,788 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-26 11:53:36,788 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1947): logDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/WALs/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:36,790 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,38971,1679831615531] 2023-03-26 11:53:36,792 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/info 2023-03-26 11:53:36,793 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-26 11:53:36,798 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:36,798 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-26 11:53:36,799 DEBUG [RS:0;jenkins-hbase4:38971] zookeeper.ZKUtil(162): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:36,801 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/rep_barrier 2023-03-26 11:53:36,802 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-26 11:53:36,803 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:36,803 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-26 11:53:36,806 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/table 2023-03-26 11:53:36,807 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-26 11:53:36,808 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:36,809 DEBUG [PEWorker-2] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740 2023-03-26 11:53:36,811 DEBUG [PEWorker-2] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740 2023-03-26 11:53:36,811 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-03-26 11:53:36,815 DEBUG [PEWorker-2] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:53:36,817 DEBUG [PEWorker-2] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-26 11:53:36,820 INFO [RS:0;jenkins-hbase4:38971] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-03-26 11:53:36,822 DEBUG [PEWorker-2] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:53:36,824 INFO [PEWorker-2] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74031042, jitterRate=0.10314849019050598}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:53:36,824 DEBUG [PEWorker-2] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-26 11:53:36,824 DEBUG [PEWorker-2] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-26 11:53:36,824 INFO [PEWorker-2] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-26 11:53:36,824 DEBUG [PEWorker-2] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-26 11:53:36,824 DEBUG [PEWorker-2] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-03-26 11:53:36,824 DEBUG [PEWorker-2] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-26 11:53:36,825 INFO [PEWorker-2] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-26 11:53:36,826 DEBUG [PEWorker-2] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-26 11:53:36,832 DEBUG [PEWorker-2] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-03-26 11:53:36,832 INFO [PEWorker-2] procedure.InitMetaProcedure(103): Going to assign meta 2023-03-26 11:53:36,842 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-03-26 11:53:36,850 INFO [RS:0;jenkins-hbase4:38971] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-03-26 11:53:36,855 INFO [RS:0;jenkins-hbase4:38971] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-03-26 11:53:36,855 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,856 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-03-26 11:53:36,860 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-03-26 11:53:36,862 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-03-26 11:53:36,869 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,869 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,870 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,871 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,871 DEBUG [RS:0;jenkins-hbase4:38971] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-03-26 11:53:36,872 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,872 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,872 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,888 INFO [RS:0;jenkins-hbase4:38971] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-03-26 11:53:36,890 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,38971,1679831615531-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:36,905 INFO [RS:0;jenkins-hbase4:38971] regionserver.Replication(203): jenkins-hbase4.apache.org,38971,1679831615531 started 2023-03-26 11:53:36,906 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,38971,1679831615531, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:38971, sessionid=0x10120ac29a00001 2023-03-26 11:53:36,906 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-03-26 11:53:36,906 DEBUG [RS:0;jenkins-hbase4:38971] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:36,906 DEBUG [RS:0;jenkins-hbase4:38971] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38971,1679831615531' 2023-03-26 11:53:36,906 DEBUG [RS:0;jenkins-hbase4:38971] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:36,907 DEBUG [RS:0;jenkins-hbase4:38971] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:36,907 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-03-26 11:53:36,907 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-03-26 11:53:36,907 DEBUG [RS:0;jenkins-hbase4:38971] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:36,907 DEBUG [RS:0;jenkins-hbase4:38971] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,38971,1679831615531' 2023-03-26 11:53:36,907 DEBUG [RS:0;jenkins-hbase4:38971] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-03-26 11:53:36,908 DEBUG [RS:0;jenkins-hbase4:38971] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-03-26 11:53:36,908 DEBUG [RS:0;jenkins-hbase4:38971] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-03-26 11:53:36,908 INFO [RS:0;jenkins-hbase4:38971] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-03-26 11:53:36,908 INFO [RS:0;jenkins-hbase4:38971] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-03-26 11:53:37,015 DEBUG [jenkins-hbase4:43599] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-03-26 11:53:37,017 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38971,1679831615531, state=OPENING 2023-03-26 11:53:37,020 INFO [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38971%2C1679831615531, suffix=, logDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/WALs/jenkins-hbase4.apache.org,38971,1679831615531, archiveDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/oldWALs, maxLogs=32 2023-03-26 11:53:37,025 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-03-26 11:53:37,026 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:37,027 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-26 11:53:37,031 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:53:37,041 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37237,DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8,DISK] 2023-03-26 11:53:37,044 INFO [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/WALs/jenkins-hbase4.apache.org,38971,1679831615531/jenkins-hbase4.apache.org%2C38971%2C1679831615531.1679831617024 2023-03-26 11:53:37,045 DEBUG [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37237,DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8,DISK]] 2023-03-26 11:53:37,216 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:37,219 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-03-26 11:53:37,222 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36228, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-03-26 11:53:37,233 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-03-26 11:53:37,233 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-03-26 11:53:37,237 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C38971%2C1679831615531.meta, suffix=.meta, logDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/WALs/jenkins-hbase4.apache.org,38971,1679831615531, archiveDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/oldWALs, maxLogs=32 2023-03-26 11:53:37,253 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:37237,DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8,DISK] 2023-03-26 11:53:37,257 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/WALs/jenkins-hbase4.apache.org,38971,1679831615531/jenkins-hbase4.apache.org%2C38971%2C1679831615531.meta.1679831617238.meta 2023-03-26 11:53:37,258 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:37237,DS-43c700c2-da22-4a0d-bbb4-5137d0110ae8,DISK]] 2023-03-26 11:53:37,258 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:53:37,260 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-03-26 11:53:37,278 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-03-26 11:53:37,283 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-03-26 11:53:37,288 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-03-26 11:53:37,288 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:37,288 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-03-26 11:53:37,288 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-03-26 11:53:37,291 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-03-26 11:53:37,293 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/info 2023-03-26 11:53:37,293 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/info 2023-03-26 11:53:37,294 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-03-26 11:53:37,295 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:37,295 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-03-26 11:53:37,296 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/rep_barrier 2023-03-26 11:53:37,296 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/rep_barrier 2023-03-26 11:53:37,297 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-03-26 11:53:37,298 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:37,298 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-03-26 11:53:37,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/table 2023-03-26 11:53:37,299 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/table 2023-03-26 11:53:37,300 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-03-26 11:53:37,301 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:37,302 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740 2023-03-26 11:53:37,305 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740 2023-03-26 11:53:37,308 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:53:37,311 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-03-26 11:53:37,312 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67372402, jitterRate=0.003927022218704224}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:53:37,312 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-03-26 11:53:37,322 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1679831617209 2023-03-26 11:53:37,339 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-03-26 11:53:37,340 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-03-26 11:53:37,341 INFO [PEWorker-1] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,38971,1679831615531, state=OPEN 2023-03-26 11:53:37,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-03-26 11:53:37,344 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-03-26 11:53:37,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-03-26 11:53:37,349 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,38971,1679831615531 in 314 msec 2023-03-26 11:53:37,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-03-26 11:53:37,354 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 508 msec 2023-03-26 11:53:37,359 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 772 msec 2023-03-26 11:53:37,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1679831617359, completionTime=-1 2023-03-26 11:53:37,359 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-03-26 11:53:37,360 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-03-26 11:53:37,421 DEBUG [hconnection-0x6bd8a8e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:37,424 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36230, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:37,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-03-26 11:53:37,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1679831677441 2023-03-26 11:53:37,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1679831737441 2023-03-26 11:53:37,441 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 81 msec 2023-03-26 11:53:37,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43599,1679831614279-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:37,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43599,1679831614279-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:37,462 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43599,1679831614279-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:37,464 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:43599, period=300000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:37,464 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-03-26 11:53:37,470 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-03-26 11:53:37,477 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-03-26 11:53:37,478 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-03-26 11:53:37,487 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-03-26 11:53:37,490 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:53:37,493 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:53:37,514 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:37,517 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5 empty. 2023-03-26 11:53:37,517 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:37,517 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-03-26 11:53:37,944 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-03-26 11:53:37,946 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => fc411bea9b9107578035998a3aeb7bf5, NAME => 'hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:53:38,358 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:38,358 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing fc411bea9b9107578035998a3aeb7bf5, disabling compactions & flushes 2023-03-26 11:53:38,358 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,359 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,359 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. after waiting 0 ms 2023-03-26 11:53:38,359 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,359 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,359 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for fc411bea9b9107578035998a3aeb7bf5: 2023-03-26 11:53:38,364 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:53:38,378 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679831618367"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831618367"}]},"ts":"1679831618367"} 2023-03-26 11:53:38,403 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:53:38,405 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:53:38,409 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831618405"}]},"ts":"1679831618405"} 2023-03-26 11:53:38,413 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-03-26 11:53:38,421 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fc411bea9b9107578035998a3aeb7bf5, ASSIGN}] 2023-03-26 11:53:38,424 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=fc411bea9b9107578035998a3aeb7bf5, ASSIGN 2023-03-26 11:53:38,426 INFO [PEWorker-5] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=fc411bea9b9107578035998a3aeb7bf5, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:53:38,577 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fc411bea9b9107578035998a3aeb7bf5, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:38,577 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679831618577"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831618577"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831618577"}]},"ts":"1679831618577"} 2023-03-26 11:53:38,582 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure fc411bea9b9107578035998a3aeb7bf5, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:53:38,743 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,744 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => fc411bea9b9107578035998a3aeb7bf5, NAME => 'hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:53:38,745 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,745 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:38,745 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,745 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,748 INFO [StoreOpener-fc411bea9b9107578035998a3aeb7bf5-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,751 DEBUG [StoreOpener-fc411bea9b9107578035998a3aeb7bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/info 2023-03-26 11:53:38,751 DEBUG [StoreOpener-fc411bea9b9107578035998a3aeb7bf5-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/info 2023-03-26 11:53:38,752 INFO [StoreOpener-fc411bea9b9107578035998a3aeb7bf5-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region fc411bea9b9107578035998a3aeb7bf5 columnFamilyName info 2023-03-26 11:53:38,753 INFO [StoreOpener-fc411bea9b9107578035998a3aeb7bf5-1] regionserver.HStore(310): Store=fc411bea9b9107578035998a3aeb7bf5/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:38,755 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,756 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,762 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:53:38,766 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:53:38,767 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened fc411bea9b9107578035998a3aeb7bf5; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=62035743, jitterRate=-0.07559539377689362}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-03-26 11:53:38,767 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for fc411bea9b9107578035998a3aeb7bf5: 2023-03-26 11:53:38,770 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5., pid=6, masterSystemTime=1679831618735 2023-03-26 11:53:38,774 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,774 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:53:38,776 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=fc411bea9b9107578035998a3aeb7bf5, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:38,777 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1679831618775"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831618775"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831618775"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831618775"}]},"ts":"1679831618775"} 2023-03-26 11:53:38,786 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-03-26 11:53:38,786 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure fc411bea9b9107578035998a3aeb7bf5, server=jenkins-hbase4.apache.org,38971,1679831615531 in 200 msec 2023-03-26 11:53:38,790 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-03-26 11:53:38,791 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=fc411bea9b9107578035998a3aeb7bf5, ASSIGN in 365 msec 2023-03-26 11:53:38,792 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:53:38,792 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831618792"}]},"ts":"1679831618792"} 2023-03-26 11:53:38,795 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-03-26 11:53:38,800 INFO [PEWorker-5] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:53:38,802 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3210 sec 2023-03-26 11:53:38,891 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-03-26 11:53:38,893 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-03-26 11:53:38,893 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:53:38,934 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-03-26 11:53:38,953 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-26 11:53:38,959 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 35 msec 2023-03-26 11:53:38,968 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-03-26 11:53:38,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-03-26 11:53:38,987 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 18 msec 2023-03-26 11:53:38,994 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-03-26 11:53:38,997 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-03-26 11:53:38,997 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 3.389sec 2023-03-26 11:53:38,999 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-03-26 11:53:39,001 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-03-26 11:53:39,001 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-03-26 11:53:39,002 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43599,1679831614279-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-03-26 11:53:39,003 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,43599,1679831614279-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-03-26 11:53:39,014 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-03-26 11:53:39,080 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x3ddd91d3 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:39,084 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51f71277, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:39,097 DEBUG [hconnection-0x286751c0-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:39,107 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36244, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:39,120 INFO [Listener at localhost/46409] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:53:39,142 INFO [Listener at localhost/46409] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=208, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=325, ProcessCount=169, AvailableMemoryMB=9634 2023-03-26 11:53:39,163 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:53:39,166 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43334, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:53:39,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:53:39,178 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:53:39,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 11:53:39,182 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:53:39,184 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:53:39,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-03-26 11:53:39,187 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:39,188 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff empty. 2023-03-26 11:53:39,190 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:39,190 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:53:39,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 11:53:39,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 11:53:39,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 11:53:39,607 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:53:39,609 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 51042bc7573c44eb24dbd242f395f3ff, NAME => 'TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:53:39,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 11:53:40,022 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:40,023 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 51042bc7573c44eb24dbd242f395f3ff, disabling compactions & flushes 2023-03-26 11:53:40,023 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,023 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,023 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. after waiting 0 ms 2023-03-26 11:53:40,023 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,023 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,023 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:40,028 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:53:40,030 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831620029"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831620029"}]},"ts":"1679831620029"} 2023-03-26 11:53:40,032 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:53:40,034 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:53:40,034 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831620034"}]},"ts":"1679831620034"} 2023-03-26 11:53:40,036 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 11:53:40,040 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, ASSIGN}] 2023-03-26 11:53:40,042 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, ASSIGN 2023-03-26 11:53:40,043 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:53:40,195 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=51042bc7573c44eb24dbd242f395f3ff, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:40,195 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831620194"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831620194"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831620194"}]},"ts":"1679831620194"} 2023-03-26 11:53:40,198 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:53:40,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 11:53:40,358 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,358 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 51042bc7573c44eb24dbd242f395f3ff, NAME => 'TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:53:40,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:53:40,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,359 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,361 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,363 DEBUG [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A 2023-03-26 11:53:40,363 DEBUG [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A 2023-03-26 11:53:40,364 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:53:40,365 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51042bc7573c44eb24dbd242f395f3ff columnFamilyName A 2023-03-26 11:53:40,365 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.HStore(310): Store=51042bc7573c44eb24dbd242f395f3ff/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:40,366 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,367 DEBUG [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B 2023-03-26 11:53:40,368 DEBUG [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B 2023-03-26 11:53:40,368 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:53:40,368 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51042bc7573c44eb24dbd242f395f3ff columnFamilyName B 2023-03-26 11:53:40,369 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.HStore(310): Store=51042bc7573c44eb24dbd242f395f3ff/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:40,369 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,371 DEBUG [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C 2023-03-26 11:53:40,371 DEBUG [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C 2023-03-26 11:53:40,371 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:53:40,372 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 51042bc7573c44eb24dbd242f395f3ff columnFamilyName C 2023-03-26 11:53:40,372 INFO [StoreOpener-51042bc7573c44eb24dbd242f395f3ff-1] regionserver.HStore(310): Store=51042bc7573c44eb24dbd242f395f3ff/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:53:40,373 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,375 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:53:40,381 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:40,384 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:53:40,385 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 51042bc7573c44eb24dbd242f395f3ff; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73919098, jitterRate=0.1014803946018219}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:53:40,385 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:40,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., pid=11, masterSystemTime=1679831620351 2023-03-26 11:53:40,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,389 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:40,389 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=51042bc7573c44eb24dbd242f395f3ff, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:40,390 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831620389"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831620389"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831620389"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831620389"}]},"ts":"1679831620389"} 2023-03-26 11:53:40,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-03-26 11:53:40,395 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 in 195 msec 2023-03-26 11:53:40,398 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-03-26 11:53:40,399 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, ASSIGN in 355 msec 2023-03-26 11:53:40,400 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:53:40,400 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831620400"}]},"ts":"1679831620400"} 2023-03-26 11:53:40,402 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 11:53:40,406 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:53:40,408 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2280 sec 2023-03-26 11:53:41,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-03-26 11:53:41,309 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-03-26 11:53:41,314 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x2678d495 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,319 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8c3f619, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,321 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,325 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36248, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,329 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:53:41,332 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:43338, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:53:41,341 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x1b886c8f to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,346 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@748f869b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,346 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x0e16d41b to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,356 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5e91d243, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,356 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x77648431 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,365 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@30eb7fb2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,366 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x7e728705 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,373 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@46192d17, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,373 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x09a575b8 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,377 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c57e1b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,378 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x1818ad0a to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,388 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49d71120, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,389 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x04fb2b82 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,421 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@59d6b6ea, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,422 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6129209e to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,426 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@d24bbb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,427 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x4b02eeb3 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:53:41,430 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63871a9d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:53:41,437 DEBUG [hconnection-0x324f527-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,438 DEBUG [hconnection-0x5926a044-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,441 DEBUG [hconnection-0x7ab648ae-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,441 DEBUG [hconnection-0x6b1a80a5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,441 DEBUG [hconnection-0x6de4daf9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,441 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36268, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,442 DEBUG [hconnection-0xe980b87-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,442 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36258, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,442 DEBUG [hconnection-0xff88c7b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,442 DEBUG [hconnection-0x461a2a49-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,445 DEBUG [hconnection-0x178bacd5-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:53:41,450 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36284, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,453 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36306, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,453 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36300, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:41,465 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36314, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,465 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36322, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,472 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36312, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,473 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36338, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:53:41,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:41,486 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:41,486 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:41,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:41,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:41,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:41,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:41,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:41,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:41,490 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:41,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:41,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:41,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:41,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:41,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:41,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:41,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:41,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:41,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:41,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:41,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:41,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:41,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:41,506 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:41,506 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:41,508 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 11:53:41,526 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:41,526 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:41,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:41,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:41,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:41,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:41,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:41,542 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:41,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=191, cellSize=8746, totalHeapSize=20104, min timestamp=1679831621562, max timestamp=1679831621612Num uniques 859; 2023-03-26 11:53:41,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=8746, suffix heap size=117064,new segment heap size=20104 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=191 2023-03-26 11:53:41,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831621601, max timestamp=1679831621612Num uniques 859; 2023-03-26 11:53:41,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:53:41,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831621601, max timestamp=1679831621618Num uniques 859; 2023-03-26 11:53:41,653 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:53:41,655 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/a5c2c8c07421425b831d8b81a0675b65 2023-03-26 11:53:41,743 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7682, totalHeapSize=17712, min timestamp=1679831621658, max timestamp=1679831621732Num uniques 1050; 2023-03-26 11:53:41,744 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48084, new segment data size=7682, suffix heap size=137168,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1050, new segment cells count=168 2023-03-26 11:53:41,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831621658, max timestamp=1679831621732Num uniques 1018; 2023-03-26 11:53:41,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-26 11:53:41,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831621658, max timestamp=1679831621732Num uniques 1018; 2023-03-26 11:53:41,750 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=6870, suffix heap size=133840,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=150 2023-03-26 11:53:41,785 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/eef069460f334e258c4565a5c28dcfb0 2023-03-26 11:53:41,813 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:41,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831681795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:41,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:41,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831681800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:41,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8092, totalHeapSize=18648, min timestamp=1679831621785, max timestamp=1679831621804Num uniques 1027; 2023-03-26 11:53:41,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:41,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831681802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:41,823 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679831621785, max timestamp=1679831621800Num uniques 1009; 2023-03-26 11:53:41,822 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47022, new segment data size=8092, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-03-26 11:53:41,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1679831621785, max timestamp=1679831621800Num uniques 1009; 2023-03-26 11:53:41,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-26 11:53:41,828 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8102, suffix heap size=132904,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=177 2023-03-26 11:53:41,885 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=10 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/2272920521104b91940bd67efcf03235 2023-03-26 11:53:41,901 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/a5c2c8c07421425b831d8b81a0675b65 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a5c2c8c07421425b831d8b81a0675b65 2023-03-26 11:53:41,914 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a5c2c8c07421425b831d8b81a0675b65, entries=150, sequenceid=10, filesize=11.6 K 2023-03-26 11:53:41,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679831621901, max timestamp=1679831621909Num uniques 1036; 2023-03-26 11:53:41,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-26 11:53:41,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679831621901, max timestamp=1679831621909Num uniques 1036; 2023-03-26 11:53:41,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-26 11:53:41,918 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-26 11:53:41,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1679831621901, max timestamp=1679831621909Num uniques 1036; 2023-03-26 11:53:41,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-03-26 11:53:41,924 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/eef069460f334e258c4565a5c28dcfb0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/eef069460f334e258c4565a5c28dcfb0 2023-03-26 11:53:41,990 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/eef069460f334e258c4565a5c28dcfb0, entries=150, sequenceid=10, filesize=11.6 K 2023-03-26 11:53:42,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/2272920521104b91940bd67efcf03235 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2272920521104b91940bd67efcf03235 2023-03-26 11:53:42,012 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2272920521104b91940bd67efcf03235, entries=150, sequenceid=10, filesize=11.6 K 2023-03-26 11:53:42,017 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=33.54 KB/34350 for 51042bc7573c44eb24dbd242f395f3ff in 509ms, sequenceid=10, compaction requested=false 2023-03-26 11:53:42,020 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 11:53:42,023 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:42,023 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:42,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:42,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:42,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:42,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37bd168a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:42,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:42,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:42,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:42,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:42,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:42,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:42,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:42,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:42,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,043 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=53.67 KB heapSize=143.39 KB 2023-03-26 11:53:42,043 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:42,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:42,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:42,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:42,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:42,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:42,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:42,055 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:42,055 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:42,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:42,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:42,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:42,055 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:42,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:42,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,064 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,091 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:42,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:42,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:42,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:42,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:42,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:42,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,094 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,094 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:42,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:42,095 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:42,095 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:42,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:42,136 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/1d8ee94d7db845509dbaac89bc9ff13a 2023-03-26 11:53:42,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831622075, max timestamp=1679831622178Num uniques 859; 2023-03-26 11:53:42,193 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:42,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831622075, max timestamp=1679831622178Num uniques 859; 2023-03-26 11:53:42,194 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:42,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831622075, max timestamp=1679831622178Num uniques 859; 2023-03-26 11:53:42,195 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:42,195 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:42,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f3059401bf68407887c87d879db9e3bd 2023-03-26 11:53:42,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:42,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:42,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:42,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:42,257 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:42,257 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:42,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:42,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:42,263 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,263 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,265 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:42,265 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:42,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:42,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:42,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:42,267 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,268 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:42,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:42,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:42,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:42,304 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831682280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:42,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:42,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:42,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:42,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:42,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831682281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:42,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831682293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831682297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831682298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,337 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:42,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:42,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:42,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:42,372 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48a74853[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:42,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:42,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,373 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:42,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:42,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:42,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:42,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:42,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:42,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:42,385 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:42,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:42,394 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:42,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:42,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:42,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:42,395 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:42,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=86 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/e05cd84161ef44e387562fc72e0203cd 2023-03-26 11:53:42,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:42,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:42,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/1d8ee94d7db845509dbaac89bc9ff13a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/1d8ee94d7db845509dbaac89bc9ff13a 2023-03-26 11:53:42,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,410 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831682409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831682411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,416 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831682415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,418 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,418 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831682415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831682415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,425 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:42,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:42,427 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:42,427 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:42,427 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:42,427 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:42,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:42,427 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:42,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:42,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:42,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:42,433 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/1d8ee94d7db845509dbaac89bc9ff13a, entries=150, sequenceid=86, filesize=11.6 K 2023-03-26 11:53:42,437 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f3059401bf68407887c87d879db9e3bd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3059401bf68407887c87d879db9e3bd 2023-03-26 11:53:42,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3059401bf68407887c87d879db9e3bd, entries=150, sequenceid=86, filesize=11.6 K 2023-03-26 11:53:42,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/e05cd84161ef44e387562fc72e0203cd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/e05cd84161ef44e387562fc72e0203cd 2023-03-26 11:53:42,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/e05cd84161ef44e387562fc72e0203cd, entries=150, sequenceid=86, filesize=11.6 K 2023-03-26 11:53:42,469 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~143.34 KB/146784, currentSize=140.89 KB/144270 for 51042bc7573c44eb24dbd242f395f3ff in 426ms, sequenceid=86, compaction requested=false 2023-03-26 11:53:42,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:42,528 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:42,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:42,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:42,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:42,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:42,569 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:42,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:42,570 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:42,570 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:42,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,582 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:42,582 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:42,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:42,583 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,583 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:42,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:42,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,584 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:42,592 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:42,592 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:42,592 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:42,592 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:42,598 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:42,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:42,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:42,600 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:42,600 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:42,600 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-26 11:53:42,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:42,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:42,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:42,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:42,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:42,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:42,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:42,620 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:42,636 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831682634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831682631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831682636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831682641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,646 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831682642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,668 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/2acc41ca7c8f424796482574547cf864 2023-03-26 11:53:42,714 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/819b69e5e9e54bc7a3bdbd56ba555c27 2023-03-26 11:53:42,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831682746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,750 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831682748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,764 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831682750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,768 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831682755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,769 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:42,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831682755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,770 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/7880f87a96104dd3a0a58356400067e3 2023-03-26 11:53:42,781 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/2acc41ca7c8f424796482574547cf864 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2acc41ca7c8f424796482574547cf864 2023-03-26 11:53:42,793 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2acc41ca7c8f424796482574547cf864, entries=150, sequenceid=122, filesize=11.6 K 2023-03-26 11:53:42,796 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/819b69e5e9e54bc7a3bdbd56ba555c27 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/819b69e5e9e54bc7a3bdbd56ba555c27 2023-03-26 11:53:42,807 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/819b69e5e9e54bc7a3bdbd56ba555c27, entries=150, sequenceid=122, filesize=11.6 K 2023-03-26 11:53:42,811 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/7880f87a96104dd3a0a58356400067e3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/7880f87a96104dd3a0a58356400067e3 2023-03-26 11:53:42,830 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/7880f87a96104dd3a0a58356400067e3, entries=150, sequenceid=122, filesize=11.6 K 2023-03-26 11:53:42,836 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=53.67 KB/54960 for 51042bc7573c44eb24dbd242f395f3ff in 236ms, sequenceid=122, compaction requested=true 2023-03-26 11:53:42,838 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:42,838 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:42,838 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:42,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:42,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:42,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:42,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:42,848 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:42,857 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@795af149[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:42,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:42,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:42,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:42,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:42,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,861 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:42,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:42,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:42,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:42,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:42,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:42,872 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:42,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:42,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:42,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:42,872 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:42,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:42,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:42,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:42,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:42,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:42,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,879 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:42,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:42,887 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:42,887 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:42,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:42,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:42,887 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:42,887 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:42,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:42,887 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:42,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:42,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:42,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:42,889 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:42,889 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:42,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:42,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:42,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:42,894 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 11:53:42,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:42,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:53:42,965 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:42,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:42,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:42,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:42,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:42,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:42,985 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-03-26 11:53:42,987 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-03-26 11:53:42,989 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:42,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:43,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:43,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831683007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:43,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:43,013 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,016 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:43,016 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831683007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:43,020 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:43,021 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831683011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:43,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831683013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:43,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831683013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:43,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:43,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:43,052 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,053 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:43,054 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:43,054 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,054 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:43,055 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:43,055 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:43,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:43,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:43,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:43,064 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:43,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f05b607[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:43,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:43,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,089 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,089 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:43,089 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:43,089 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:43,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:43,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:43,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:43,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:43,096 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:43,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:43,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:43,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:43,096 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:43,099 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,099 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,100 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,104 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:43,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:43,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:43,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,118 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:43,118 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:43,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831683120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,126 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831683126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831683128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,138 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831683137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831683137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,218 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:43,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:43,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:43,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:43,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:43,245 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:43,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:43,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:43,256 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,256 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,258 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,258 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:43,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:43,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:43,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:43,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:43,268 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:43,268 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:43,268 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,268 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:43,268 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:43,268 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:43,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:43,275 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:43,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30d3f42b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:43,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:43,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,283 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:43,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:43,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:43,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:43,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:43,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:43,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:43,290 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:43,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:43,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:43,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:43,290 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:43,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,296 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,303 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,303 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,303 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:43,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:43,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,304 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:43,304 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:43,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831683325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,333 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831683332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831683333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,342 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831683341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,343 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831683343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,404 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:43,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:43,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:43,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:43,420 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:43,420 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:43,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:43,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,424 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,424 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,426 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,426 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:43,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,427 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:43,431 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:43,431 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:43,431 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:43,431 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,432 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:43,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/80e9faa422544ba6bf505d6a596b4b33 2023-03-26 11:53:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:43,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:43,444 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,444 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:43,444 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:43,445 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:43,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:43,448 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:43,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42c4d7b2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:43,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:43,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:43,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:43,455 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,455 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:43,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:43,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:43,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:43,469 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:43,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:43,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:43,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:43,469 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:43,472 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,472 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,472 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,483 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,483 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,483 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,483 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,484 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:43,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:43,484 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:43,484 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:43,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,486 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,491 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/dad4d1c050db437189e8fc7311347412 2023-03-26 11:53:43,537 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=135 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/78fa7fc924ac443daa9506db962a4a45 2023-03-26 11:53:43,550 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/80e9faa422544ba6bf505d6a596b4b33 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/80e9faa422544ba6bf505d6a596b4b33 2023-03-26 11:53:43,562 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/80e9faa422544ba6bf505d6a596b4b33, entries=150, sequenceid=135, filesize=11.8 K 2023-03-26 11:53:43,565 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/dad4d1c050db437189e8fc7311347412 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/dad4d1c050db437189e8fc7311347412 2023-03-26 11:53:43,582 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/dad4d1c050db437189e8fc7311347412, entries=150, sequenceid=135, filesize=11.8 K 2023-03-26 11:53:43,586 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:43,588 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/78fa7fc924ac443daa9506db962a4a45 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/78fa7fc924ac443daa9506db962a4a45 2023-03-26 11:53:43,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:43,593 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:43,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:43,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:43,596 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:43,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:43,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:43,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,601 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,601 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:43,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:43,603 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:43,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:43,603 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:43,603 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:43,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:43,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:43,607 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/78fa7fc924ac443daa9506db962a4a45, entries=150, sequenceid=135, filesize=11.8 K 2023-03-26 11:53:43,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:43,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:43,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:43,614 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 51042bc7573c44eb24dbd242f395f3ff in 649ms, sequenceid=135, compaction requested=true 2023-03-26 11:53:43,615 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:43,619 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,619 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:43,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c3ab61a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:43,623 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:43,623 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:43,626 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:43,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:43,627 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:43,627 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:43,627 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:43,628 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:43,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,630 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:43,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,631 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:43,635 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:43,635 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:43,635 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:43,635 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:43,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:43,636 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:43,637 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:43,640 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:43,640 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,640 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-26 11:53:43,640 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/eef069460f334e258c4565a5c28dcfb0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3059401bf68407887c87d879db9e3bd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/819b69e5e9e54bc7a3bdbd56ba555c27, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/dad4d1c050db437189e8fc7311347412] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=46.6 K 2023-03-26 11:53:43,640 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:43,640 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:43,641 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:43,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:43,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,645 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting eef069460f334e258c4565a5c28dcfb0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679831621482 2023-03-26 11:53:43,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,644 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,649 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,653 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f3059401bf68407887c87d879db9e3bd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679831622023 2023-03-26 11:53:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,654 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 819b69e5e9e54bc7a3bdbd56ba555c27, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679831622240 2023-03-26 11:53:43,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,654 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,655 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dad4d1c050db437189e8fc7311347412, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679831622632 2023-03-26 11:53:43,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,658 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:43,658 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,658 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a5c2c8c07421425b831d8b81a0675b65, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/1d8ee94d7db845509dbaac89bc9ff13a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2acc41ca7c8f424796482574547cf864, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/80e9faa422544ba6bf505d6a596b4b33] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=46.6 K 2023-03-26 11:53:43,663 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a5c2c8c07421425b831d8b81a0675b65, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679831621482 2023-03-26 11:53:43,664 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1d8ee94d7db845509dbaac89bc9ff13a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679831622023 2023-03-26 11:53:43,665 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2acc41ca7c8f424796482574547cf864, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679831622240 2023-03-26 11:53:43,668 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,668 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:43,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:43,668 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,668 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 80e9faa422544ba6bf505d6a596b4b33, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679831622632 2023-03-26 11:53:43,668 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,673 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,671 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:43,673 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:43,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,678 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831683658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831683671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831683674, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,701 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831683679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,713 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831683680, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,717 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/f92d1e1bc38145489d17a2760ac6d4cd 2023-03-26 11:53:43,737 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#13 average throughput is 0.47 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:43,744 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#14 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:43,773 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:43,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:43,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:43,806 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831683793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:43,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:43,806 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:43,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:43,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:43,808 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,808 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:43,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,811 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:43,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:43,812 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:43,812 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:43,812 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:43,812 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:43,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:43,820 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,820 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:43,820 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:43,820 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831683817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,820 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:43,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:43,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831683817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,823 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831683817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:43,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831683817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,825 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,825 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:43,829 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ecd8014[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:43,829 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:43,829 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,830 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:43,831 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:43,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:43,833 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:43,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:43,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:43,839 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:43,839 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:43,839 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:43,839 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:43,839 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:43,843 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,843 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,843 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,844 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:43,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:43,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:43,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:43,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:43,857 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:43,857 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:43,957 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:43,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:43,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:43,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:43,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:43,967 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:43,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:43,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:43,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,974 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:43,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:43,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:43,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:43,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:43,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:43,980 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:43,980 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:43,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:43,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:43,984 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:43,984 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:43,984 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:43,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:43,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:43,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:43,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:43,987 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:43,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:43,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:43,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:43,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:43,991 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1905b4a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:43,991 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:43,991 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:43,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:43,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:43,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:43,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:43,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:43,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:43,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:43,998 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:43,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:43,998 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:44,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,005 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831684009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:44,013 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:44,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,017 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:44,017 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:44,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831684025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831684022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,032 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831684027, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,033 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831684028, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,117 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:44,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:44,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:44,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:44,126 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:44,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:44,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:44,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,135 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:44,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:44,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,138 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:44,141 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:44,142 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:44,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:44,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:44,148 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:44,148 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,148 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:44,148 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:44,148 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:44,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:44,151 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,152 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,152 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:44,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658d658d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:44,154 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:44,154 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:44,157 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:44,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:44,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,161 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:44,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:44,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:44,188 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:44,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:44,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:44,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:44,188 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:44,190 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,190 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,190 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:44,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,198 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:44,199 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:44,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:44,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/59d13b30ce2a4fc6b8047f05c9ac43e0 2023-03-26 11:53:44,218 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9d3bfdae65184c01839e86b7d82205d9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9d3bfdae65184c01839e86b7d82205d9 2023-03-26 11:53:44,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/fc84b9a174124525962bc7c430fad71e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fc84b9a174124525962bc7c430fad71e 2023-03-26 11:53:44,248 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into 9d3bfdae65184c01839e86b7d82205d9(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:44,248 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into fc84b9a174124525962bc7c430fad71e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:44,248 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:44,248 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:44,248 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=12, startTime=1679831623626; duration=0sec 2023-03-26 11:53:44,248 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=12, startTime=1679831623627; duration=0sec 2023-03-26 11:53:44,249 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:44,249 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:44,249 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:44,252 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47694 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:44,252 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:44,252 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,252 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2272920521104b91940bd67efcf03235, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/e05cd84161ef44e387562fc72e0203cd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/7880f87a96104dd3a0a58356400067e3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/78fa7fc924ac443daa9506db962a4a45] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=46.6 K 2023-03-26 11:53:44,253 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2272920521104b91940bd67efcf03235, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=10, earliestPutTs=1679831621482 2023-03-26 11:53:44,254 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting e05cd84161ef44e387562fc72e0203cd, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=86, earliestPutTs=1679831622023 2023-03-26 11:53:44,255 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 7880f87a96104dd3a0a58356400067e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679831622240 2023-03-26 11:53:44,255 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 78fa7fc924ac443daa9506db962a4a45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679831622632 2023-03-26 11:53:44,267 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/bdbcc0f187d74a16a399901f0023cb6d 2023-03-26 11:53:44,277 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/f92d1e1bc38145489d17a2760ac6d4cd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/f92d1e1bc38145489d17a2760ac6d4cd 2023-03-26 11:53:44,283 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#17 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:44,302 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:44,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:44,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:44,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:44,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:44,310 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:44,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,311 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/f92d1e1bc38145489d17a2760ac6d4cd, entries=150, sequenceid=156, filesize=11.8 K 2023-03-26 11:53:44,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:44,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:44,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:44,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/59d13b30ce2a4fc6b8047f05c9ac43e0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/59d13b30ce2a4fc6b8047f05c9ac43e0 2023-03-26 11:53:44,329 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831684319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831684329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831684330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,336 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,336 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:44,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831684334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,337 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:44,337 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:44,337 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:44,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:44,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:44,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:44,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/59d13b30ce2a4fc6b8047f05c9ac43e0, entries=150, sequenceid=156, filesize=11.8 K 2023-03-26 11:53:44,341 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,341 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:44,341 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:44,341 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,343 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:44,343 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:44,343 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,343 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:44,344 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:44,345 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/bdbcc0f187d74a16a399901f0023cb6d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/bdbcc0f187d74a16a399901f0023cb6d 2023-03-26 11:53:44,346 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:44,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:44,349 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cbe8540[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:44,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:44,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:44,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:44,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:44,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:44,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831684343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,353 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:44,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:44,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,357 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/bdbcc0f187d74a16a399901f0023cb6d, entries=150, sequenceid=156, filesize=11.8 K 2023-03-26 11:53:44,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,358 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 51042bc7573c44eb24dbd242f395f3ff in 718ms, sequenceid=156, compaction requested=false 2023-03-26 11:53:44,358 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:44,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:44,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:44,358 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:44,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:44,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:44,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:44,359 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:44,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:44,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:44,397 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:44,397 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:44,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,460 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-26 11:53:44,461 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-03-26 11:53:44,497 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:44,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:44,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:44,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:44,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:44,504 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:44,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:44,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:44,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:44,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:44,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:44,512 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:44,512 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:44,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,513 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:44,514 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:44,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:44,515 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,517 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:44,517 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:53:44,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:44,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:44,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:44,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:44,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:44,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:44,574 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/5004121d7e7c48c79dc01d8346371303 2023-03-26 11:53:44,654 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/49767752109c41b09db64719412c4502 2023-03-26 11:53:44,716 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=170 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/5c040779725e4349b413a310cbef50ff 2023-03-26 11:53:44,725 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/b7e0c15f8f474080a444ae3ef65ae482 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/b7e0c15f8f474080a444ae3ef65ae482 2023-03-26 11:53:44,740 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/5004121d7e7c48c79dc01d8346371303 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5004121d7e7c48c79dc01d8346371303 2023-03-26 11:53:44,749 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into b7e0c15f8f474080a444ae3ef65ae482(size=11.9 K), total size for store is 23.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:44,750 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:44,750 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=12, startTime=1679831623627; duration=0sec 2023-03-26 11:53:44,750 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5004121d7e7c48c79dc01d8346371303, entries=150, sequenceid=170, filesize=11.8 K 2023-03-26 11:53:44,750 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:44,762 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/49767752109c41b09db64719412c4502 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/49767752109c41b09db64719412c4502 2023-03-26 11:53:44,775 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/49767752109c41b09db64719412c4502, entries=150, sequenceid=170, filesize=11.8 K 2023-03-26 11:53:44,779 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/5c040779725e4349b413a310cbef50ff as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/5c040779725e4349b413a310cbef50ff 2023-03-26 11:53:44,797 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/5c040779725e4349b413a310cbef50ff, entries=150, sequenceid=170, filesize=11.8 K 2023-03-26 11:53:44,799 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 51042bc7573c44eb24dbd242f395f3ff in 282ms, sequenceid=170, compaction requested=true 2023-03-26 11:53:44,799 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:44,799 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,800 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:44,801 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:44,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,806 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:44,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:44,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e3173e3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:44,809 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:44,809 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:44,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:44,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:44,811 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,811 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:44,812 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:44,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:44,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:44,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:44,819 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:44,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:44,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:44,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:44,819 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:44,821 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,821 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,821 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,835 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,835 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,835 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:44,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,836 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:44,836 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:44,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:44,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,860 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:44,861 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:44,861 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:44,861 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:44,861 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:44,861 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:44,861 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:44,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:44,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831684908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679831624891, max timestamp=1679831624910Num uniques 859; 2023-03-26 11:53:44,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 11:53:44,915 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679831624891, max timestamp=1679831624910Num uniques 859; 2023-03-26 11:53:44,916 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831624892, max timestamp=1679831624910Num uniques 859; 2023-03-26 11:53:44,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:53:44,917 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 11:53:44,937 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:44,942 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:44,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:44,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:44,944 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:44,944 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:44,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:44,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,954 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831684947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831684948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:44,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,957 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831684949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:44,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:44,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:44,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:44,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831684950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:44,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:44,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:44,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:44,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:44,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:44,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:44,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:44,966 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:44,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:44,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:44,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:44,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:44,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,970 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,970 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:44,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:44,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1df82164[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:44,971 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:44,971 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,972 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:44,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:44,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:44,976 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:44,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:44,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:44,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:44,986 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:44,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:44,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:44,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:44,986 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:44,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:44,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:44,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:44,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:44,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:44,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,999 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,005 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:45,005 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,005 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:45,007 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:45,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,007 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:45,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,007 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:45,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,017 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831685016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,061 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831685059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831685064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,070 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831685066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831685067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,108 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:45,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:45,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:45,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:45,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:45,115 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:45,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:45,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:45,117 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,117 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,118 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:45,120 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:45,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,120 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:45,121 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:45,121 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:45,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:45,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:45,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:45,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:45,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:45,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:45,131 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:45,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@13578e5b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:45,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:45,136 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,137 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:45,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:45,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,139 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:45,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:45,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:45,147 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:45,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:45,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:45,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:45,147 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:45,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,153 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,160 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,160 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,160 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:45,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:45,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:45,169 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:45,169 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:45,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831685219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,269 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:45,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:45,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831685269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:45,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831685274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:45,281 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:45,281 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:45,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:45,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831685277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,284 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,285 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831685279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:45,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:45,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,286 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:45,287 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:45,287 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:45,287 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:45,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:45,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:45,288 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,289 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:45,289 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:45,289 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:45,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:45,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:45,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:45,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,291 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:45,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3beb9df4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:45,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:45,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:45,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:45,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,301 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:45,301 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:45,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:45,309 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:45,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:45,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:45,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:45,309 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:45,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:45,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:45,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:45,323 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:45,323 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:45,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,336 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/d08bb0da4c9c409eacdf7220a1b99775 2023-03-26 11:53:45,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/b54c26eaf7ef447da38539bc67296271 2023-03-26 11:53:45,421 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/3805cb941aa3458da5b2b542565c90db 2023-03-26 11:53:45,423 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:45,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:45,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:45,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:45,434 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:45,434 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:45,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:45,437 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,437 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/d08bb0da4c9c409eacdf7220a1b99775 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d08bb0da4c9c409eacdf7220a1b99775 2023-03-26 11:53:45,439 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,440 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:45,442 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,442 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:45,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:45,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:45,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:45,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:45,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:45,446 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,446 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:45,446 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:45,446 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:45,446 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,449 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d08bb0da4c9c409eacdf7220a1b99775, entries=150, sequenceid=181, filesize=11.8 K 2023-03-26 11:53:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:45,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,454 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/b54c26eaf7ef447da38539bc67296271 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b54c26eaf7ef447da38539bc67296271 2023-03-26 11:53:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:45,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e505f01[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:45,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:45,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:45,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:45,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:45,460 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:45,461 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,461 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,461 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,464 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b54c26eaf7ef447da38539bc67296271, entries=150, sequenceid=181, filesize=11.8 K 2023-03-26 11:53:45,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:45,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:45,466 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:45,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:45,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:45,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:45,466 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:45,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/3805cb941aa3458da5b2b542565c90db as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3805cb941aa3458da5b2b542565c90db 2023-03-26 11:53:45,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,470 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,478 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3805cb941aa3458da5b2b542565c90db, entries=150, sequenceid=181, filesize=11.8 K 2023-03-26 11:53:45,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,480 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,479 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=138.26 KB/141575 for 51042bc7573c44eb24dbd242f395f3ff in 619ms, sequenceid=181, compaction requested=true 2023-03-26 11:53:45,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,480 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:45,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,480 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:45,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,482 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:45,482 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:45,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:45,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:45,482 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:45,482 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:45,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,487 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:45,487 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:45,489 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:45,490 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:45,491 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,491 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,491 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fc84b9a174124525962bc7c430fad71e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/59d13b30ce2a4fc6b8047f05c9ac43e0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/49767752109c41b09db64719412c4502, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b54c26eaf7ef447da38539bc67296271] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=47.2 K 2023-03-26 11:53:45,491 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9d3bfdae65184c01839e86b7d82205d9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/f92d1e1bc38145489d17a2760ac6d4cd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5004121d7e7c48c79dc01d8346371303, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d08bb0da4c9c409eacdf7220a1b99775] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=47.2 K 2023-03-26 11:53:45,492 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fc84b9a174124525962bc7c430fad71e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679831622632 2023-03-26 11:53:45,493 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9d3bfdae65184c01839e86b7d82205d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679831622632 2023-03-26 11:53:45,494 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 59d13b30ce2a4fc6b8047f05c9ac43e0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679831623006 2023-03-26 11:53:45,495 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 49767752109c41b09db64719412c4502, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679831623671 2023-03-26 11:53:45,495 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f92d1e1bc38145489d17a2760ac6d4cd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679831623006 2023-03-26 11:53:45,496 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5004121d7e7c48c79dc01d8346371303, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679831623671 2023-03-26 11:53:45,496 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b54c26eaf7ef447da38539bc67296271, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1679831624856 2023-03-26 11:53:45,498 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d08bb0da4c9c409eacdf7220a1b99775, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1679831624856 2023-03-26 11:53:45,532 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:45,533 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=144.97 KB heapSize=404.77 KB 2023-03-26 11:53:45,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:45,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:45,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:45,533 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:45,533 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:45,534 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:45,544 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#24 average throughput is 1.09 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:45,551 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#25 average throughput is 0.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:45,582 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:45,595 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831685585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,595 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831685590, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,596 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:45,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831685593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831685593, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,603 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831685595, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,613 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:45,621 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/45f203c3ad97418981535bfd97316596 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/45f203c3ad97418981535bfd97316596 2023-03-26 11:53:45,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:45,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:45,636 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:45,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,638 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/fb7eaed762374bb293949879721e7d89 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/fb7eaed762374bb293949879721e7d89 2023-03-26 11:53:45,641 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into 45f203c3ad97418981535bfd97316596(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:45,641 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:45,641 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=12, startTime=1679831625482; duration=0sec 2023-03-26 11:53:45,641 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:45,641 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:45,644 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:45,644 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:45,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,646 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,656 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48282 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:45,657 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:45,657 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,657 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/b7e0c15f8f474080a444ae3ef65ae482, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/bdbcc0f187d74a16a399901f0023cb6d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/5c040779725e4349b413a310cbef50ff, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3805cb941aa3458da5b2b542565c90db] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=47.2 K 2023-03-26 11:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:45,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:45,659 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b7e0c15f8f474080a444ae3ef65ae482, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=135, earliestPutTs=1679831622632 2023-03-26 11:53:45,659 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:45,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:45,662 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:45,662 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,663 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:45,664 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:45,665 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bdbcc0f187d74a16a399901f0023cb6d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679831623006 2023-03-26 11:53:45,665 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,665 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:45,665 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:45,666 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:45,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:45,666 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5c040779725e4349b413a310cbef50ff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=170, earliestPutTs=1679831623671 2023-03-26 11:53:45,667 WARN [DataStreamer for file /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9bc80a05d11f4816a625999112fb1c96] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-26 11:53:45,667 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3805cb941aa3458da5b2b542565c90db, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1679831624856 2023-03-26 11:53:45,668 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into fb7eaed762374bb293949879721e7d89(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:45,668 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:45,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9bc80a05d11f4816a625999112fb1c96 2023-03-26 11:53:45,668 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=12, startTime=1679831625480; duration=0sec 2023-03-26 11:53:45,668 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:45,674 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,674 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:45,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:45,683 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@727fa8ae[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:45,683 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:45,683 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,686 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,686 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:45,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:45,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,691 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:45,693 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:45,698 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:45,698 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:45,698 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:45,698 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:45,698 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:45,700 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,700 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,700 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,702 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831685705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831685705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831685705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,710 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#27 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:45,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,714 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:45,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:45,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,714 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,714 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:45,715 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:45,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,740 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/83ad5d097b0447d4a55e7a696a2e44b8 2023-03-26 11:53:45,815 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:45,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:45,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:45,821 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:45,821 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:45,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:45,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,826 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,826 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,827 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,827 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:45,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:45,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,830 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:45,832 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:45,832 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:45,832 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:45,837 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,837 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:45,837 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:45,837 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:45,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:45,839 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:45,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:45,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5df5c5ec[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:45,843 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:45,843 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,845 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:45,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:45,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:45,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:45,847 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:45,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:45,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:45,852 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:45,852 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:45,852 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:45,853 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:45,853 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:45,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,862 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,871 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:45,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:45,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:45,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:45,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,875 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:45,875 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:45,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:45,908 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831685908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,913 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,913 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831685910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:45,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831685910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,976 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:45,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:45,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:45,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:45,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:45,983 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:45,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:45,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:45,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,986 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:45,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:45,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:45,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:45,989 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:45,989 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:45,989 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:45,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:45,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:45,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:45,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:45,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:45,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:45,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:45,994 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,994 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:45,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:45,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:45,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a96f79b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:45,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,997 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:45,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:45,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:45,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:46,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:46,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:46,010 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:46,010 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:46,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,016 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:46,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:46,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,020 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,021 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:46,025 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:46,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,099 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831686097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,100 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831686098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,125 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:46,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:46,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:46,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:46,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:46,130 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:46,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:46,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:46,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:46,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,135 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:46,135 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:46,135 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:46,138 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,138 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:46,138 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:46,138 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:46,141 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:46,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16920f9f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:46,146 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:46,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,148 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/88fc80278ae74115aa6eb9b84cdcf6b0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/88fc80278ae74115aa6eb9b84cdcf6b0 2023-03-26 11:53:46,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:46,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:46,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:46,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:46,151 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:46,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:46,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:46,159 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:46,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:46,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:46,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:46,159 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:46,164 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into 88fc80278ae74115aa6eb9b84cdcf6b0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:46,164 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:46,164 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=12, startTime=1679831625482; duration=0sec 2023-03-26 11:53:46,165 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:46,165 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,165 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,165 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,175 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,176 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:46,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:46,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:46,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:46,191 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=217 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/af702c6ebc44475db23c9e3075ae7048 2023-03-26 11:53:46,199 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9bc80a05d11f4816a625999112fb1c96 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9bc80a05d11f4816a625999112fb1c96 2023-03-26 11:53:46,208 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9bc80a05d11f4816a625999112fb1c96, entries=200, sequenceid=217, filesize=14.1 K 2023-03-26 11:53:46,212 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/83ad5d097b0447d4a55e7a696a2e44b8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/83ad5d097b0447d4a55e7a696a2e44b8 2023-03-26 11:53:46,216 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831686212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,217 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831686215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831686217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/83ad5d097b0447d4a55e7a696a2e44b8, entries=150, sequenceid=217, filesize=11.8 K 2023-03-26 11:53:46,236 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/af702c6ebc44475db23c9e3075ae7048 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/af702c6ebc44475db23c9e3075ae7048 2023-03-26 11:53:46,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/af702c6ebc44475db23c9e3075ae7048, entries=150, sequenceid=217, filesize=11.8 K 2023-03-26 11:53:46,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~144.97 KB/148445, heapSize ~404.73 KB/414440, currentSize=40.25 KB/41220 for 51042bc7573c44eb24dbd242f395f3ff in 720ms, sequenceid=217, compaction requested=false 2023-03-26 11:53:46,254 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:46,282 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:46,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:46,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:46,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:46,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:46,296 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:46,296 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,296 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:46,296 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:46,297 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,297 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,298 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:46,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:46,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:46,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:46,300 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,301 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:46,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:46,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:46,313 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 11:53:46,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:46,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:46,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:46,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:46,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:46,315 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:46,365 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/d117cd150e89412d80063cbf0ae926da 2023-03-26 11:53:46,405 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/cef668f78fc04575848a7fe76a1c7fe7 2023-03-26 11:53:46,486 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=229 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/26bf5221f981491ab898c48b068f8fa0 2023-03-26 11:53:46,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/d117cd150e89412d80063cbf0ae926da as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d117cd150e89412d80063cbf0ae926da 2023-03-26 11:53:46,509 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d117cd150e89412d80063cbf0ae926da, entries=150, sequenceid=229, filesize=11.8 K 2023-03-26 11:53:46,516 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/cef668f78fc04575848a7fe76a1c7fe7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/cef668f78fc04575848a7fe76a1c7fe7 2023-03-26 11:53:46,525 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/cef668f78fc04575848a7fe76a1c7fe7, entries=150, sequenceid=229, filesize=11.8 K 2023-03-26 11:53:46,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/26bf5221f981491ab898c48b068f8fa0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/26bf5221f981491ab898c48b068f8fa0 2023-03-26 11:53:46,536 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/26bf5221f981491ab898c48b068f8fa0, entries=150, sequenceid=229, filesize=11.8 K 2023-03-26 11:53:46,538 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 51042bc7573c44eb24dbd242f395f3ff in 224ms, sequenceid=229, compaction requested=true 2023-03-26 11:53:46,538 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:46,538 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,538 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:46,541 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:46,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,546 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:46,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:46,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19a7ff3b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:46,552 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:46,552 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,553 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,554 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:46,554 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:46,554 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:46,554 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,554 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:46,558 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:46,558 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,558 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,558 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:46,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:46,566 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:46,566 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:46,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:46,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:46,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:46,566 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:46,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,570 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,570 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,583 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:46,583 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:46,584 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:46,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:46,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:46,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:46,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:46,689 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:46,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:46,690 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:46,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:46,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:46,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:46,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:46,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:46,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:46,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:46,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:46,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:46,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:46,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:46,698 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,698 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:46,698 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:46,699 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:46,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:46,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:46,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:46,700 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,701 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:46,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8e9346[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:46,705 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:46,705 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,706 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,707 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:46,707 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:46,707 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:46,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:46,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:46,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:46,713 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:46,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:46,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:46,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:46,713 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:46,719 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,719 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,719 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,721 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,726 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,734 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:46,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:46,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:46,734 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:46,734 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:46,735 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:46,735 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:46,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:46,736 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,736 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,736 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,737 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:46,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,738 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:46,738 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:46,762 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/3c1e143cfaa24b8781c3cbd8370b2fae 2023-03-26 11:53:46,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831626753, max timestamp=1679831626808Num uniques 859; 2023-03-26 11:53:46,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:46,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831626753, max timestamp=1679831626808Num uniques 859; 2023-03-26 11:53:46,810 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:46,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831626753, max timestamp=1679831626808Num uniques 859; 2023-03-26 11:53:46,811 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:46,838 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:46,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:46,844 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:46,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:46,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:46,846 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:46,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:46,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:46,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,847 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,848 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,848 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,849 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:46,849 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:46,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,850 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:46,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831686844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831686845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831686847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:46,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:46,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:46,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:46,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:46,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:46,860 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,860 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:46,860 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:46,860 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:46,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:46,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:46,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67299ba3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:46,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:46,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:46,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:46,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:46,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:46,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:46,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:46,872 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:46,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:46,876 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:46,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:46,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:46,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:46,877 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:46,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:46,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:46,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:46,882 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:46,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:46,884 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:46,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:46,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:46,885 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:46,885 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:46,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:46,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:46,958 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831686955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,959 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831686956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,959 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:46,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831686956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:46,985 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:46,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:46,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:46,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:46,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:46,996 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:46,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:46,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:46,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:46,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:46,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:47,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:47,016 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:47,017 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:47,028 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,028 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:47,028 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:47,028 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:47,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:47,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:47,031 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,031 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,033 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,033 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:47,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61c19e2e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:47,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:47,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:47,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:47,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,038 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:47,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:47,041 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:47,041 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:47,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:47,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:47,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:47,041 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:47,045 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,045 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,045 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:47,055 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,055 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:47,055 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:47,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,104 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831687101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831687105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,156 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:47,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:47,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831687161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831687162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:47,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:47,165 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:47,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:47,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:47,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,167 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831687166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,168 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,168 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:47,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:47,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:47,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:47,176 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,176 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:47,176 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:47,176 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:47,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:47,179 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,180 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:47,183 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4739fb0a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:47,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:47,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,184 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f6fe0202237f4bc18aa3fd36c7dbb563 2023-03-26 11:53:47,185 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:47,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:47,185 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:47,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:47,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:47,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:47,194 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:47,195 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:47,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:47,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:47,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:47,195 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:47,199 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,199 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,199 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,201 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,205 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,205 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:47,205 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:47,206 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:47,206 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:47,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:47,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,306 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:47,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:47,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:47,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:47,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:47,316 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:47,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:47,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:47,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:47,321 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:47,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,322 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:47,340 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:47,340 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,340 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:47,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:47,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:47,348 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,348 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:47,348 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:47,348 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:47,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:47,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:47,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:47,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:47,354 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59334408[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:47,354 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:47,354 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,355 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:47,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:47,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:47,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:47,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:47,368 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:47,368 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:47,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,371 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:47,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:47,376 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:47,376 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,377 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:47,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831687464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,468 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831687466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831687473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,476 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:47,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:47,480 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:47,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:47,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:47,492 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:47,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:47,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,497 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:47,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:47,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:47,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:47,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:47,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:47,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:47,503 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:47,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@726c380e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:47,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:47,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:47,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:47,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:47,510 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,511 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:47,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:47,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:47,515 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:47,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:47,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:47,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:47,516 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:47,517 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,517 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,518 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:47,529 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:47,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,530 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:47,530 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:47,624 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=240 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/2a5878a31d8740fa8b274cb1e877b99c 2023-03-26 11:53:47,630 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:47,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:47,634 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/3c1e143cfaa24b8781c3cbd8370b2fae as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/3c1e143cfaa24b8781c3cbd8370b2fae 2023-03-26 11:53:47,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:47,637 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:47,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:47,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:47,638 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,638 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,639 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:47,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,641 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,642 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:47,644 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/3c1e143cfaa24b8781c3cbd8370b2fae, entries=200, sequenceid=240, filesize=14.1 K 2023-03-26 11:53:47,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:47,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:47,649 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,650 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:47,650 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:47,650 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:47,650 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f6fe0202237f4bc18aa3fd36c7dbb563 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f6fe0202237f4bc18aa3fd36c7dbb563 2023-03-26 11:53:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:47,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:47,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:47,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,656 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f6fe0202237f4bc18aa3fd36c7dbb563, entries=150, sequenceid=240, filesize=11.8 K 2023-03-26 11:53:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,660 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/2a5878a31d8740fa8b274cb1e877b99c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2a5878a31d8740fa8b274cb1e877b99c 2023-03-26 11:53:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb1f883[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:47,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:47,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,665 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:47,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:47,667 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:47,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:47,670 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:47,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:47,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:47,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:47,670 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:47,672 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2a5878a31d8740fa8b274cb1e877b99c, entries=150, sequenceid=240, filesize=11.8 K 2023-03-26 11:53:47,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,673 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,673 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:47,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:47,674 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51042bc7573c44eb24dbd242f395f3ff in 940ms, sequenceid=240, compaction requested=true 2023-03-26 11:53:47,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,674 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:47,674 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:47,675 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:47,674 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:47,675 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,677 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,677 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 53198 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:47,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:47,677 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:47,678 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,678 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/fb7eaed762374bb293949879721e7d89, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9bc80a05d11f4816a625999112fb1c96, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d117cd150e89412d80063cbf0ae926da, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/3c1e143cfaa24b8781c3cbd8370b2fae] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=52.0 K 2023-03-26 11:53:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:47,679 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting fb7eaed762374bb293949879721e7d89, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1679831624856 2023-03-26 11:53:47,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,681 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9bc80a05d11f4816a625999112fb1c96, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679831624945 2023-03-26 11:53:47,681 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:47,681 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:47,681 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,681 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/45f203c3ad97418981535bfd97316596, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/83ad5d097b0447d4a55e7a696a2e44b8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/cef668f78fc04575848a7fe76a1c7fe7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f6fe0202237f4bc18aa3fd36c7dbb563] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=47.3 K 2023-03-26 11:53:47,681 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d117cd150e89412d80063cbf0ae926da, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679831625536 2023-03-26 11:53:47,683 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 45f203c3ad97418981535bfd97316596, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1679831624856 2023-03-26 11:53:47,683 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3c1e143cfaa24b8781c3cbd8370b2fae, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1679831626725 2023-03-26 11:53:47,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:47,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,684 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:47,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:47,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:47,684 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:47,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:47,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:47,684 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 83ad5d097b0447d4a55e7a696a2e44b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679831624947 2023-03-26 11:53:47,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,685 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:47,685 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:47,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:47,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,685 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting cef668f78fc04575848a7fe76a1c7fe7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679831625536 2023-03-26 11:53:47,686 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f6fe0202237f4bc18aa3fd36c7dbb563, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1679831626725 2023-03-26 11:53:47,698 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#36 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:47,702 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#37 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:47,720 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/a3ac063ee20a4feea145fb20012ce1dd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a3ac063ee20a4feea145fb20012ce1dd 2023-03-26 11:53:47,735 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into a3ac063ee20a4feea145fb20012ce1dd(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:47,735 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:47,735 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=12, startTime=1679831627674; duration=0sec 2023-03-26 11:53:47,735 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:47,735 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:47,738 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:47,738 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:47,738 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,738 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/88fc80278ae74115aa6eb9b84cdcf6b0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/af702c6ebc44475db23c9e3075ae7048, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/26bf5221f981491ab898c48b068f8fa0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2a5878a31d8740fa8b274cb1e877b99c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=47.3 K 2023-03-26 11:53:47,739 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 88fc80278ae74115aa6eb9b84cdcf6b0, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1679831624856 2023-03-26 11:53:47,739 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting af702c6ebc44475db23c9e3075ae7048, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=217, earliestPutTs=1679831624947 2023-03-26 11:53:47,740 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 26bf5221f981491ab898c48b068f8fa0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=229, earliestPutTs=1679831625536 2023-03-26 11:53:47,740 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/67f6c230d10f48fc9f34b23fe14eab6a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/67f6c230d10f48fc9f34b23fe14eab6a 2023-03-26 11:53:47,742 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2a5878a31d8740fa8b274cb1e877b99c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1679831626725 2023-03-26 11:53:47,754 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into 67f6c230d10f48fc9f34b23fe14eab6a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:47,754 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:47,754 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=12, startTime=1679831627674; duration=0sec 2023-03-26 11:53:47,754 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:47,757 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#38 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:47,785 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:47,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:47,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:47,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:47,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:47,789 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:47,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:47,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:47,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,792 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:47,792 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:47,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:47,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:47,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:47,794 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:47,794 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:47,794 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:47,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:47,795 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:47,795 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:47,810 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/2a4074cc920c4b199c68d013e32edbc0 2023-03-26 11:53:47,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:47,974 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:47,988 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831687985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831687990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:47,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:47,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831687991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,092 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831688091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831688093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831688097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,181 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/43519555bd10480aa63adf3afb09dbd4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/43519555bd10480aa63adf3afb09dbd4 2023-03-26 11:53:48,195 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into 43519555bd10480aa63adf3afb09dbd4(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:48,196 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:48,196 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=12, startTime=1679831627675; duration=0sec 2023-03-26 11:53:48,196 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:48,231 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/971bb98440944220af6a16f3f6ca829a 2023-03-26 11:53:48,260 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=279 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/54104fc6815b4dbeae4ab83a146665c3 2023-03-26 11:53:48,275 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/2a4074cc920c4b199c68d013e32edbc0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2a4074cc920c4b199c68d013e32edbc0 2023-03-26 11:53:48,289 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2a4074cc920c4b199c68d013e32edbc0, entries=150, sequenceid=279, filesize=11.9 K 2023-03-26 11:53:48,290 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/971bb98440944220af6a16f3f6ca829a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/971bb98440944220af6a16f3f6ca829a 2023-03-26 11:53:48,303 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831688296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,306 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831688297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831688305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,309 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/971bb98440944220af6a16f3f6ca829a, entries=150, sequenceid=279, filesize=11.9 K 2023-03-26 11:53:48,316 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/54104fc6815b4dbeae4ab83a146665c3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/54104fc6815b4dbeae4ab83a146665c3 2023-03-26 11:53:48,337 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/54104fc6815b4dbeae4ab83a146665c3, entries=150, sequenceid=279, filesize=11.9 K 2023-03-26 11:53:48,339 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=46.96 KB/48090 for 51042bc7573c44eb24dbd242f395f3ff in 544ms, sequenceid=279, compaction requested=false 2023-03-26 11:53:48,339 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:48,339 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:48,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:48,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:48,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:48,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:48,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,343 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:48,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,344 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:48,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:48,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:48,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17d8b1d4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:48,346 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:48,347 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:48,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:48,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:48,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:48,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:48,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:48,351 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:48,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:48,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:48,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:48,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:48,355 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:48,355 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:48,355 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:48,355 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:48,355 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:48,358 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,358 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:48,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:48,359 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:48,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:48,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:48,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:48,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,364 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:48,369 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:48,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:48,369 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:48,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:48,369 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:48,370 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:48,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:48,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:48,470 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:48,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:48,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:48,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:48,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:48,478 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:48,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:48,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,479 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:48,481 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:48,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:48,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:48,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:48,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:48,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:48,482 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:48,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:48,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:48,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:48,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:48,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:48,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:48,489 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:48,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:48,611 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:48,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:48,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831628628, max timestamp=1679831628644Num uniques 859; 2023-03-26 11:53:48,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831628628, max timestamp=1679831628644Num uniques 859; 2023-03-26 11:53:48,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:48,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831628628, max timestamp=1679831628644Num uniques 859; 2023-03-26 11:53:48,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:48,647 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:48,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831688685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36284 deadline: 1679831688685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831688689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831688790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831688793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831688800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:48,943 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/a1b448ec12114067ac3036bbbe430d75 2023-03-26 11:53:48,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:48,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831688994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831688997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831689006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831689109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,112 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4160 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:49,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831689118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,121 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:49,302 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831689302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831689304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,312 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36258 deadline: 1679831689310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,362 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/2557c5aba8e14b6bbf22e462103fb472 2023-03-26 11:53:49,388 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=290 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/9ab9159d974f45a5b59a4532b55f9706 2023-03-26 11:53:49,398 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/a1b448ec12114067ac3036bbbe430d75 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a1b448ec12114067ac3036bbbe430d75 2023-03-26 11:53:49,406 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a1b448ec12114067ac3036bbbe430d75, entries=150, sequenceid=290, filesize=11.9 K 2023-03-26 11:53:49,407 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/2557c5aba8e14b6bbf22e462103fb472 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/2557c5aba8e14b6bbf22e462103fb472 2023-03-26 11:53:49,414 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/2557c5aba8e14b6bbf22e462103fb472, entries=150, sequenceid=290, filesize=11.9 K 2023-03-26 11:53:49,417 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/9ab9159d974f45a5b59a4532b55f9706 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9ab9159d974f45a5b59a4532b55f9706 2023-03-26 11:53:49,425 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9ab9159d974f45a5b59a4532b55f9706, entries=150, sequenceid=290, filesize=11.9 K 2023-03-26 11:53:49,426 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 51042bc7573c44eb24dbd242f395f3ff in 937ms, sequenceid=290, compaction requested=true 2023-03-26 11:53:49,426 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:49,426 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:49,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:49,429 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:49,429 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:49,432 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,434 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,434 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:49,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:49,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:49,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5408b423[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:49,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:49,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,439 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:49,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:49,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:49,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:49,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:49,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:49,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:49,443 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:49,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:49,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:49,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:49,444 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:49,445 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,445 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,445 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:49,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:49,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:49,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:49,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:49,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:49,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,451 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:49,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:49,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:49,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:49,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:49,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:49,454 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:49,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:49,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:49,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:49,454 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:49,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:49,455 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,455 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,455 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:49,455 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:49,455 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:49,455 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:49,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:49,555 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:49,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:49,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:49,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:49,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:49,559 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:49,559 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:49,559 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:49,559 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:49,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:49,562 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:49,562 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:49,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:49,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:49,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:49,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:49,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:49,564 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:49,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:49,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:49,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:49,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:49,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:49,568 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-26 11:53:49,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:49,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:49,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:49,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:49,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:49,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:49,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:49,808 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:49,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831689827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831689830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,836 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831689833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36312 deadline: 1679831689934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831689935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:49,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831689937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:49,983 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/add24bcbf53d46f99f1ebf20172bdd3c 2023-03-26 11:53:50,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:50,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831690139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:50,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831690143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:50,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831690143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,415 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/c727e1deab9f4db0925377020c949387 2023-03-26 11:53:50,436 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/3e8049e2bf4441f5af6587b8303ad334 2023-03-26 11:53:50,444 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:50,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831690442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,448 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/add24bcbf53d46f99f1ebf20172bdd3c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/add24bcbf53d46f99f1ebf20172bdd3c 2023-03-26 11:53:50,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:50,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831690446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,451 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:50,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831690450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,456 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/add24bcbf53d46f99f1ebf20172bdd3c, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 11:53:50,459 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/c727e1deab9f4db0925377020c949387 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c727e1deab9f4db0925377020c949387 2023-03-26 11:53:50,465 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c727e1deab9f4db0925377020c949387, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 11:53:50,467 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/3e8049e2bf4441f5af6587b8303ad334 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3e8049e2bf4441f5af6587b8303ad334 2023-03-26 11:53:50,473 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3e8049e2bf4441f5af6587b8303ad334, entries=150, sequenceid=326, filesize=11.9 K 2023-03-26 11:53:50,474 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for 51042bc7573c44eb24dbd242f395f3ff in 906ms, sequenceid=326, compaction requested=true 2023-03-26 11:53:50,474 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:50,475 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:50,475 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:50,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:50,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:50,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:50,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:50,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:50,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47375ae4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:50,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:50,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,489 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:50,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:50,489 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,489 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:50,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:50,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:50,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:50,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:50,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:50,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:50,496 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:50,496 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:50,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:50,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:50,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:50,496 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:50,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:50,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:50,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:50,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:50,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:50,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:50,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,505 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:50,505 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:50,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:50,506 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:50,505 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:50,506 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:50,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:50,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:50,506 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:50,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:50,506 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:50,506 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:50,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:50,507 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:50,506 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:50,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,607 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:50,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:50,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:50,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:50,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:50,613 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:50,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:50,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:50,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:50,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:50,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:50,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:50,615 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:50,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:50,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:50,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:50,618 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:50,618 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:50,618 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:50,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:50,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:50,621 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:50,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:50,638 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/b84472cc8d314af5855dc9e84e59d79a 2023-03-26 11:53:50,954 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:50,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:50,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679831630959, max timestamp=1679831630991Num uniques 859; 2023-03-26 11:53:50,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679831630959, max timestamp=1679831630991Num uniques 859; 2023-03-26 11:53:50,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 11:53:50,993 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=359, cellSize=16435, totalHeapSize=37576, min timestamp=1679831630959, max timestamp=1679831630991Num uniques 859; 2023-03-26 11:53:50,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 11:53:50,994 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=16435, suffix heap size=117064,new segment heap size=37576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=359 2023-03-26 11:53:51,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831691021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,026 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831691022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,030 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831691025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,054 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/6e14b32f110a47c6a352e7219e432aa7 2023-03-26 11:53:51,070 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/243fe59387f64b6ab2567fe314596f8b 2023-03-26 11:53:51,080 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/b84472cc8d314af5855dc9e84e59d79a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b84472cc8d314af5855dc9e84e59d79a 2023-03-26 11:53:51,090 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b84472cc8d314af5855dc9e84e59d79a, entries=150, sequenceid=336, filesize=11.9 K 2023-03-26 11:53:51,092 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/6e14b32f110a47c6a352e7219e432aa7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/6e14b32f110a47c6a352e7219e432aa7 2023-03-26 11:53:51,103 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/6e14b32f110a47c6a352e7219e432aa7, entries=150, sequenceid=336, filesize=11.9 K 2023-03-26 11:53:51,105 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/243fe59387f64b6ab2567fe314596f8b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/243fe59387f64b6ab2567fe314596f8b 2023-03-26 11:53:51,120 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/243fe59387f64b6ab2567fe314596f8b, entries=150, sequenceid=336, filesize=11.9 K 2023-03-26 11:53:51,121 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51042bc7573c44eb24dbd242f395f3ff in 500ms, sequenceid=336, compaction requested=true 2023-03-26 11:53:51,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:51,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:51,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,123 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:51,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57d2d761[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:51,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:51,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:51,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:51,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:51,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:51,130 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=147.60 KB heapSize=406.08 KB 2023-03-26 11:53:51,130 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:51,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,132 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:51,132 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:51,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:51,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:51,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:51,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:51,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:51,140 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:51,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:51,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:51,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:51,140 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:51,143 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,143 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,143 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,146 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:51,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:51,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:51,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,150 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:51,150 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:51,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,151 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/b0e4ded7b79f44c4bcca631589e1b65a 2023-03-26 11:53:51,171 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/b4d372ab5ebd48569db27664fe914e69 2023-03-26 11:53:51,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/528cc522d1cd41edad95512ed89d0b89 2023-03-26 11:53:51,192 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/b0e4ded7b79f44c4bcca631589e1b65a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b0e4ded7b79f44c4bcca631589e1b65a 2023-03-26 11:53:51,192 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831691188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,193 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831691188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831691192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b0e4ded7b79f44c4bcca631589e1b65a, entries=200, sequenceid=371, filesize=14.3 K 2023-03-26 11:53:51,207 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/b4d372ab5ebd48569db27664fe914e69 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b4d372ab5ebd48569db27664fe914e69 2023-03-26 11:53:51,214 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b4d372ab5ebd48569db27664fe914e69, entries=150, sequenceid=371, filesize=11.9 K 2023-03-26 11:53:51,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/528cc522d1cd41edad95512ed89d0b89 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/528cc522d1cd41edad95512ed89d0b89 2023-03-26 11:53:51,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/528cc522d1cd41edad95512ed89d0b89, entries=150, sequenceid=371, filesize=11.9 K 2023-03-26 11:53:51,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~406.03 KB/415776, currentSize=40.25 KB/41220 for 51042bc7573c44eb24dbd242f395f3ff in 94ms, sequenceid=371, compaction requested=true 2023-03-26 11:53:51,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:51,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:51,224 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:53:51,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:51,225 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:51,225 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:53:51,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 75816 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:53:51,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:51,228 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,228 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/67f6c230d10f48fc9f34b23fe14eab6a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2a4074cc920c4b199c68d013e32edbc0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a1b448ec12114067ac3036bbbe430d75, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/add24bcbf53d46f99f1ebf20172bdd3c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b84472cc8d314af5855dc9e84e59d79a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b0e4ded7b79f44c4bcca631589e1b65a] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=74.0 K 2023-03-26 11:53:51,229 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73376 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:53:51,229 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 67f6c230d10f48fc9f34b23fe14eab6a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1679831626725 2023-03-26 11:53:51,230 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:51,230 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,230 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a3ac063ee20a4feea145fb20012ce1dd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/971bb98440944220af6a16f3f6ca829a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/2557c5aba8e14b6bbf22e462103fb472, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c727e1deab9f4db0925377020c949387, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/6e14b32f110a47c6a352e7219e432aa7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b4d372ab5ebd48569db27664fe914e69] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=71.7 K 2023-03-26 11:53:51,230 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2a4074cc920c4b199c68d013e32edbc0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679831626830 2023-03-26 11:53:51,230 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a3ac063ee20a4feea145fb20012ce1dd, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1679831626725 2023-03-26 11:53:51,231 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a1b448ec12114067ac3036bbbe430d75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679831627972 2023-03-26 11:53:51,231 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 971bb98440944220af6a16f3f6ca829a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679831626830 2023-03-26 11:53:51,231 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting add24bcbf53d46f99f1ebf20172bdd3c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679831628671 2023-03-26 11:53:51,231 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2557c5aba8e14b6bbf22e462103fb472, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679831627972 2023-03-26 11:53:51,232 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b84472cc8d314af5855dc9e84e59d79a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1679831629806 2023-03-26 11:53:51,232 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c727e1deab9f4db0925377020c949387, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679831628671 2023-03-26 11:53:51,232 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b0e4ded7b79f44c4bcca631589e1b65a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679831630996 2023-03-26 11:53:51,233 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6e14b32f110a47c6a352e7219e432aa7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1679831629806 2023-03-26 11:53:51,233 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b4d372ab5ebd48569db27664fe914e69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679831630996 2023-03-26 11:53:51,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:51,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:51,253 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#54 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:51,262 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#55 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:51,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:51,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:51,278 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:51,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:51,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:51,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,290 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,291 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,298 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:51,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:51,301 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:51,301 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:51,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:51,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:51,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:51,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:51,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:51,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:51,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:51,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:51,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:51,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:51,311 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:51,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:51,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:51,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d69a5e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:51,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:51,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,318 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:51,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:51,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,320 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:51,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:51,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:51,326 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:51,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:51,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:51,327 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:51,327 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:51,329 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,329 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,334 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,338 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:51,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,338 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:51,338 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:51,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831631344, max timestamp=1679831631373Num uniques 859; 2023-03-26 11:53:51,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831631344, max timestamp=1679831631373Num uniques 859; 2023-03-26 11:53:51,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:51,375 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:51,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831631344, max timestamp=1679831631373Num uniques 859; 2023-03-26 11:53:51,377 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:51,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831691406, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831691412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831691416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,438 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:51,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:51,443 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:51,445 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:51,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:51,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:51,446 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,446 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,448 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:51,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,450 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:51,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:51,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:51,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:51,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:51,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,455 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,455 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:51,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:51,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@146e9f6b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:51,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:51,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,470 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:51,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:51,470 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,470 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:51,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:51,482 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:51,482 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:51,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:51,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:51,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:51,482 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:51,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,489 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:51,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,490 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:51,490 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:51,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,516 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831691515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831691522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,526 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831691522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,590 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:51,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:51,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:51,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:51,600 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:51,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:51,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:51,601 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,601 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,603 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,604 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:51,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,604 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:51,605 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:51,605 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:51,605 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,605 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:51,605 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:51,605 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:51,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:51,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:51,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:51,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:51,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69f30778[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:51,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:51,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,610 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,610 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:51,610 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:51,610 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,610 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:51,610 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:51,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:51,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:51,624 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:51,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:51,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:51,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:51,624 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:51,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,629 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,633 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,633 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:51,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:51,636 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:51,636 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:51,704 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/df879f42fef04e7ba28d9eb8ece9cac8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/df879f42fef04e7ba28d9eb8ece9cac8 2023-03-26 11:53:51,712 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into df879f42fef04e7ba28d9eb8ece9cac8(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:51,712 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:51,712 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=10, startTime=1679831631224; duration=0sec 2023-03-26 11:53:51,712 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:51,712 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:53:51,715 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73376 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:53:51,715 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:51,715 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,715 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/43519555bd10480aa63adf3afb09dbd4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/54104fc6815b4dbeae4ab83a146665c3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9ab9159d974f45a5b59a4532b55f9706, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3e8049e2bf4441f5af6587b8303ad334, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/243fe59387f64b6ab2567fe314596f8b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/528cc522d1cd41edad95512ed89d0b89] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=71.7 K 2023-03-26 11:53:51,716 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 43519555bd10480aa63adf3afb09dbd4, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=240, earliestPutTs=1679831626725 2023-03-26 11:53:51,716 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 54104fc6815b4dbeae4ab83a146665c3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=279, earliestPutTs=1679831626830 2023-03-26 11:53:51,717 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9ab9159d974f45a5b59a4532b55f9706, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=290, earliestPutTs=1679831627972 2023-03-26 11:53:51,718 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3e8049e2bf4441f5af6587b8303ad334, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1679831628671 2023-03-26 11:53:51,718 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 243fe59387f64b6ab2567fe314596f8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=336, earliestPutTs=1679831629806 2023-03-26 11:53:51,719 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 528cc522d1cd41edad95512ed89d0b89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679831630996 2023-03-26 11:53:51,720 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831691718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,721 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/a96ee4b34a1948a8b41cb04f0793b08f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a96ee4b34a1948a8b41cb04f0793b08f 2023-03-26 11:53:51,729 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into a96ee4b34a1948a8b41cb04f0793b08f(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:51,730 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:51,730 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=10, startTime=1679831631224; duration=0sec 2023-03-26 11:53:51,730 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:51,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831691729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:51,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831691729, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,736 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:51,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:51,739 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/8da865cfb8ac4f8b81041a110ead3bc1 2023-03-26 11:53:51,745 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:51,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:51,747 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:51,747 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,747 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:51,747 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:51,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,750 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:51,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,753 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:51,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:51,753 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#57 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:51,756 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:51,756 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:51,756 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,756 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:51,757 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:51,757 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:51,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:51,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,759 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,760 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:51,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:51,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@285bbcf8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:51,763 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:51,763 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,766 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,766 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:51,768 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:51,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:51,770 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,770 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:51,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:51,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:51,773 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:51,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:51,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:51,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:51,773 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:51,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,775 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,777 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,779 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:51,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:51,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:51,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,780 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:51,780 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:51,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,789 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/1c3c9fc7b7f5402287accf307880bf80 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/1c3c9fc7b7f5402287accf307880bf80 2023-03-26 11:53:51,796 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into 1c3c9fc7b7f5402287accf307880bf80(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:51,796 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:51,796 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=10, startTime=1679831631224; duration=0sec 2023-03-26 11:53:51,796 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:51,880 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:51,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:51,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:51,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:51,884 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:51,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:51,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:51,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,887 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,887 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,888 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,888 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:51,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,889 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:51,889 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:51,889 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:51,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:51,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:51,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:51,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:51,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:51,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:51,890 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:51,890 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:51,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:51,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,892 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19eec7c7[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:51,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:51,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:51,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:51,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:51,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:51,902 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:51,902 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:51,902 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:51,902 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:51,902 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:51,904 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,904 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:51,905 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,905 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:51,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,907 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:51,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:51,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:51,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:51,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:51,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:51,912 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:51,912 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:51,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:51,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:51,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:51,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,012 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,023 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831692021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,026 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,026 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:52,027 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,027 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,032 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,032 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,032 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,032 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,032 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:52,032 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,032 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,032 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,032 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,033 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,033 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:52,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831692036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:52,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@791b9656[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:52,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:52,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831692039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,039 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:52,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:52,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:52,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:52,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:52,047 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:52,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:52,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:52,048 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:52,048 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:52,049 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,049 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,049 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,051 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:52,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:52,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,057 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:52,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:52,057 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:52,157 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,166 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,168 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,168 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,168 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,169 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:52,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,179 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,180 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/305775a453d4424f8edb940038641015 2023-03-26 11:53:52,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,181 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,181 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,181 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:52,181 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,185 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,185 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:52,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:52,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@82296ca[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:52,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:52,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,190 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:52,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:52,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:52,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:52,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:52,195 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:52,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:52,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:52,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:52,195 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:52,196 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,196 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,197 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:52,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:52,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,204 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:52,204 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,204 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:52,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,211 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=381 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/9426cfafe2a24ec7b8c19d121805259d 2023-03-26 11:53:52,217 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/8da865cfb8ac4f8b81041a110ead3bc1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8da865cfb8ac4f8b81041a110ead3bc1 2023-03-26 11:53:52,222 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8da865cfb8ac4f8b81041a110ead3bc1, entries=150, sequenceid=381, filesize=11.9 K 2023-03-26 11:53:52,223 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/305775a453d4424f8edb940038641015 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/305775a453d4424f8edb940038641015 2023-03-26 11:53:52,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/305775a453d4424f8edb940038641015, entries=150, sequenceid=381, filesize=11.9 K 2023-03-26 11:53:52,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/9426cfafe2a24ec7b8c19d121805259d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9426cfafe2a24ec7b8c19d121805259d 2023-03-26 11:53:52,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9426cfafe2a24ec7b8c19d121805259d, entries=150, sequenceid=381, filesize=11.9 K 2023-03-26 11:53:52,239 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 51042bc7573c44eb24dbd242f395f3ff in 935ms, sequenceid=381, compaction requested=false 2023-03-26 11:53:52,239 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:52,305 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,309 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,312 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:52,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,313 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-26 11:53:52,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:52,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:52,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:52,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:52,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:52,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:52,326 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/422126d619504ebeb7dd7ea13226e03a 2023-03-26 11:53:52,349 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/d9440c4bec0a4ef8af4f5048928afcf5 2023-03-26 11:53:52,361 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=419 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/240dea58dca342e68fcd4fe1931a66c1 2023-03-26 11:53:52,367 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/422126d619504ebeb7dd7ea13226e03a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/422126d619504ebeb7dd7ea13226e03a 2023-03-26 11:53:52,372 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/422126d619504ebeb7dd7ea13226e03a, entries=150, sequenceid=419, filesize=11.9 K 2023-03-26 11:53:52,373 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/d9440c4bec0a4ef8af4f5048928afcf5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d9440c4bec0a4ef8af4f5048928afcf5 2023-03-26 11:53:52,379 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d9440c4bec0a4ef8af4f5048928afcf5, entries=150, sequenceid=419, filesize=11.9 K 2023-03-26 11:53:52,380 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/240dea58dca342e68fcd4fe1931a66c1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/240dea58dca342e68fcd4fe1931a66c1 2023-03-26 11:53:52,385 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/240dea58dca342e68fcd4fe1931a66c1, entries=150, sequenceid=419, filesize=11.9 K 2023-03-26 11:53:52,386 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=0 B/0 for 51042bc7573c44eb24dbd242f395f3ff in 73ms, sequenceid=419, compaction requested=true 2023-03-26 11:53:52,386 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:52,386 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,387 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,389 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:52,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7deb474a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:52,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:52,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:52,393 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:52,393 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:52,396 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:52,399 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:52,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:52,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:52,400 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:52,400 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:52,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,402 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:52,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:52,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,409 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:52,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,409 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:52,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,514 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,517 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,517 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,517 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,517 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,518 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,518 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:52,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,519 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,519 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,519 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:52,519 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,521 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:52,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:52,524 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125477cf[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:52,524 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:52,524 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:52,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:52,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:52,527 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:52,531 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:52,531 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:52,531 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:52,531 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:52,531 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:52,531 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:52,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,535 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:52,542 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:52,542 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:52,542 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:52,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:52,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:52,543 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:52,543 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,549 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:52,549 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,549 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:52,549 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:52,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:52,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679831632551, max timestamp=1679831632579Num uniques 859; 2023-03-26 11:53:52,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679831632551, max timestamp=1679831632579Num uniques 859; 2023-03-26 11:53:52,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 11:53:52,581 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 11:53:52,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=409, cellSize=18725, totalHeapSize=42776, min timestamp=1679831632551, max timestamp=1679831632579Num uniques 859; 2023-03-26 11:53:52,582 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=18725, suffix heap size=117064,new segment heap size=42776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=409 2023-03-26 11:53:52,584 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/740fae4105834d729ddedd22e98ced74 2023-03-26 11:53:52,607 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831692601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,608 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831692606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,609 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831692607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,639 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/b97b8e7569494b469c018275728fd063 2023-03-26 11:53:52,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,655 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,665 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,666 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/4ab198ae0760447f947e71dd0470f346 2023-03-26 11:53:52,668 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,671 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,671 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,673 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,679 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/740fae4105834d729ddedd22e98ced74 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/740fae4105834d729ddedd22e98ced74 2023-03-26 11:53:52,686 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:52,687 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,691 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/740fae4105834d729ddedd22e98ced74, entries=100, sequenceid=430, filesize=9.5 K 2023-03-26 11:53:52,692 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/b97b8e7569494b469c018275728fd063 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b97b8e7569494b469c018275728fd063 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:52,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b97b8e7569494b469c018275728fd063, entries=100, sequenceid=430, filesize=9.5 K 2023-03-26 11:53:52,701 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/4ab198ae0760447f947e71dd0470f346 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/4ab198ae0760447f947e71dd0470f346 2023-03-26 11:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:52,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3758eaf5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:52,704 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:52,704 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:52,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:52,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:52,712 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:52,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/4ab198ae0760447f947e71dd0470f346, entries=100, sequenceid=430, filesize=9.5 K 2023-03-26 11:53:52,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,715 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51042bc7573c44eb24dbd242f395f3ff in 173ms, sequenceid=430, compaction requested=true 2023-03-26 11:53:52,716 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:52,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:52,716 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:52,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:52,716 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:52,716 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:52,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:52,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:52,716 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:52,716 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:52,716 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:52,716 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:52,717 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:52,718 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:52,718 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:52,718 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,718 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/df879f42fef04e7ba28d9eb8ece9cac8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8da865cfb8ac4f8b81041a110ead3bc1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/422126d619504ebeb7dd7ea13226e03a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/740fae4105834d729ddedd22e98ced74] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=45.8 K 2023-03-26 11:53:52,718 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:52,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:52,718 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting df879f42fef04e7ba28d9eb8ece9cac8, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679831630996 2023-03-26 11:53:52,718 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:52,719 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=147.60 KB heapSize=401.39 KB 2023-03-26 11:53:52,719 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a96ee4b34a1948a8b41cb04f0793b08f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/305775a453d4424f8edb940038641015, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d9440c4bec0a4ef8af4f5048928afcf5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b97b8e7569494b469c018275728fd063] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=45.8 K 2023-03-26 11:53:52,719 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8da865cfb8ac4f8b81041a110ead3bc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1679831631138 2023-03-26 11:53:52,720 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 422126d619504ebeb7dd7ea13226e03a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679831631401 2023-03-26 11:53:52,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:52,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:52,720 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:52,720 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:52,721 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:52,721 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:52,721 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,721 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,721 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,721 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 740fae4105834d729ddedd22e98ced74, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679831632540 2023-03-26 11:53:52,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,722 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a96ee4b34a1948a8b41cb04f0793b08f, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679831630996 2023-03-26 11:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,723 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 305775a453d4424f8edb940038641015, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1679831631138 2023-03-26 11:53:52,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,723 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d9440c4bec0a4ef8af4f5048928afcf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679831631401 2023-03-26 11:53:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,724 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b97b8e7569494b469c018275728fd063, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679831632540 2023-03-26 11:53:52,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,724 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,728 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,728 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,728 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:52,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:52,728 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,729 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:52,729 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:52,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,729 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,729 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:52,730 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,748 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#67 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:52,748 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#68 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:52,757 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9dbe29acff6b4500989a2ab43a42bf4b 2023-03-26 11:53:52,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831692768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831692797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831692797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,803 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/1183f985188c48149ff2a7a1f933e266 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1183f985188c48149ff2a7a1f933e266 2023-03-26 11:53:52,818 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into 1183f985188c48149ff2a7a1f933e266(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:52,818 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:52,818 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=12, startTime=1679831632716; duration=0sec 2023-03-26 11:53:52,818 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:52,818 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:53:52,820 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:53:52,820 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:52,820 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,821 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/1c3c9fc7b7f5402287accf307880bf80, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9426cfafe2a24ec7b8c19d121805259d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/240dea58dca342e68fcd4fe1931a66c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/4ab198ae0760447f947e71dd0470f346] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=45.8 K 2023-03-26 11:53:52,821 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1c3c9fc7b7f5402287accf307880bf80, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1679831630996 2023-03-26 11:53:52,822 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9426cfafe2a24ec7b8c19d121805259d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=381, earliestPutTs=1679831631138 2023-03-26 11:53:52,823 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 240dea58dca342e68fcd4fe1931a66c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=419, earliestPutTs=1679831631401 2023-03-26 11:53:52,823 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 4ab198ae0760447f947e71dd0470f346, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679831632540 2023-03-26 11:53:52,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/3531697cc60246ef8f99e040d11c62ec 2023-03-26 11:53:52,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,836 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,841 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,842 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,849 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#71 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:52,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,853 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:52,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:52,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b50be12[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:52,858 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:52,858 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:52,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:52,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:52,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:52,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:52,866 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:52,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:52,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:52,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:52,866 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:52,874 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/586bdfa32db9424897d148ad90cb761b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/586bdfa32db9424897d148ad90cb761b 2023-03-26 11:53:52,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:52,878 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:52,880 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,882 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into 586bdfa32db9424897d148ad90cb761b(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:52,882 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:52,882 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=12, startTime=1679831632716; duration=0sec 2023-03-26 11:53:52,882 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:52,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:52,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,883 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,883 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:52,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:52,900 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831692896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831692898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,903 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:52,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831692899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,983 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:52,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:52,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:52,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:52,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:52,988 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:52,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:52,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:52,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,989 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:52,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:52,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:52,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,994 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,994 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:52,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:52,994 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:52,995 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:52,995 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:52,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:52,995 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,995 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:52,996 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:52,996 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:52,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:52,998 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:52,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:52,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c266d25[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,009 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,015 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,015 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,015 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,015 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,015 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,024 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,024 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,024 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831693101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,105 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831693104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831693104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,124 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,128 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,130 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,130 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,131 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,131 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,131 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,131 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,131 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,132 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,132 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,132 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,133 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dfd8961[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,136 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,137 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,137 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,140 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36322 deadline: 1679831693144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,147 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8195 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,148 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,148 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,149 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,154 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,155 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,155 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,155 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,155 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,158 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:36306 deadline: 1679831693158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,160 DEBUG [Thread-149] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8211 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:53:53,203 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/b25dab08206f4d13ac4dc992c08d9139 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b25dab08206f4d13ac4dc992c08d9139 2023-03-26 11:53:53,210 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into b25dab08206f4d13ac4dc992c08d9139(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:53,210 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:53,210 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=12, startTime=1679831632716; duration=0sec 2023-03-26 11:53:53,210 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:53,255 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,258 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/a405458e159b458d98244fb511210a71 2023-03-26 11:53:53,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,260 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,261 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,263 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,263 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,264 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,265 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,265 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,265 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9dbe29acff6b4500989a2ab43a42bf4b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9dbe29acff6b4500989a2ab43a42bf4b 2023-03-26 11:53:53,265 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,265 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,268 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,270 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d02c369[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,270 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,270 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,271 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,272 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,272 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,273 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9dbe29acff6b4500989a2ab43a42bf4b, entries=150, sequenceid=465, filesize=11.9 K 2023-03-26 11:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/3531697cc60246ef8f99e040d11c62ec as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/3531697cc60246ef8f99e040d11c62ec 2023-03-26 11:53:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,279 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,279 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,279 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/3531697cc60246ef8f99e040d11c62ec, entries=150, sequenceid=465, filesize=11.9 K 2023-03-26 11:53:53,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/a405458e159b458d98244fb511210a71 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/a405458e159b458d98244fb511210a71 2023-03-26 11:53:53,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,291 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/a405458e159b458d98244fb511210a71, entries=150, sequenceid=465, filesize=11.9 K 2023-03-26 11:53:53,293 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~421.27 KB/431376, currentSize=40.25 KB/41220 for 51042bc7573c44eb24dbd242f395f3ff in 573ms, sequenceid=465, compaction requested=false 2023-03-26 11:53:53,293 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:53,293 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,293 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,293 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,293 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,294 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,294 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,294 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,294 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,294 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,294 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,294 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,293 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,394 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,398 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,400 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,405 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,405 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,405 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,405 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:53,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:53,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:53,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:53,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:53,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:53,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:53,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,426 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,430 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@638c54f5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,430 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,430 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,436 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831633421, max timestamp=1679831633435Num uniques 859; 2023-03-26 11:53:53,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:53,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831633421, max timestamp=1679831633435Num uniques 859; 2023-03-26 11:53:53,438 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:53,439 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831633421, max timestamp=1679831633435Num uniques 859; 2023-03-26 11:53:53,440 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,444 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,444 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,447 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,447 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,447 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,451 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,451 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,451 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831693468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831693469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831693469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,552 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,554 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,555 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,558 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,558 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,559 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,559 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,560 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,560 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,560 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,560 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,561 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,561 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,561 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,561 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,561 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,562 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,562 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,563 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,564 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cfc42e4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,566 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,566 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,567 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,571 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,571 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831693573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831693573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,576 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,576 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,576 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831693574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,581 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,585 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,586 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,586 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,586 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,586 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,586 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,586 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,586 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,586 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,586 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,686 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,689 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,691 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,692 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,692 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,693 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,693 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,693 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,694 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d9b6e9b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,696 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,702 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,702 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,702 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,702 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,702 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,703 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,703 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,704 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,706 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,707 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,713 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,713 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,713 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,713 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,778 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831693777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,779 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831693778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,784 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:53,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831693783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,813 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,816 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,817 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,817 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,817 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,817 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,818 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,818 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,820 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,820 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,820 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,821 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,821 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,821 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,821 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,822 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,822 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,822 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,822 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,822 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,824 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5659fdca[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,826 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,826 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,827 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,828 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,828 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,833 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7 2023-03-26 11:53:53,833 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:53,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,837 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,837 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,839 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,840 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,840 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,840 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,847 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,847 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,847 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,849 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ba0a3dfc0a454959a06b57a3c732ea50 2023-03-26 11:53:53,851 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,851 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,851 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,852 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,852 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:53,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,852 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,852 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,852 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,852 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,952 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:53,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:53,955 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:53,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:53,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:53,957 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:53,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:53,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:53,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,959 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,959 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:53,960 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,961 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:53,961 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:53,961 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:53,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:53,962 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,962 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:53,962 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:53,962 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:53,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:53,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:53,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57f33008[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:53,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:53,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,977 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:53,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:53,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:53,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:53,981 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:53,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:53,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:53,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:53,981 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:53,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:53,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:53,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:53,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:53,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:53,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:53,991 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:53,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:53,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:53,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:53,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:53,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:53,992 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:53,992 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:53,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:53,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:53,992 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:53,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:53,992 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831694081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,084 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831694082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,089 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831694086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,093 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:54,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:54,097 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:54,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:54,098 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:54,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:54,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:54,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:54,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,101 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:54,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:54,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,102 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,102 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:54,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:54,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:54,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:54,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:54,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:54,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:54,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:54,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:54,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:54,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:54,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:54,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:54,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:54,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6926b370[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:54,113 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:54,113 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:54,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:54,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:54,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:54,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:54,119 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:54,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:54,121 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:54,121 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:54,121 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:54,121 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:54,121 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:54,122 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:54,123 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,123 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,123 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,123 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:54,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:54,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:54,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:54,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:54,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:54,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:54,129 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:54,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:54,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:54,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:54,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:54,129 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:54,130 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:54,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:54,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:54,230 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:54,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:54,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:54,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:54,233 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:54,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:54,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:54,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:54,234 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,234 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,235 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:54,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:54,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,237 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:54,237 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:54,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:54,237 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:54,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:54,240 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:54,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:54,241 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,241 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f1fe779[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:54,242 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:54,242 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,243 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:54,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:54,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:54,244 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:54,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:54,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:54,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:54,247 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:54,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:54,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:54,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:54,247 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:54,249 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,249 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,249 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:54,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:54,249 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:54,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:54,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,256 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:54,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:54,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:54,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:54,257 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:54,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:54,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,257 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:54,257 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:54,257 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:54,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:54,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:54,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/c4c18af8af9d4e0b84b79188a10f4e3d 2023-03-26 11:53:54,276 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7 2023-03-26 11:53:54,282 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7, entries=100, sequenceid=478, filesize=9.5 K 2023-03-26 11:53:54,283 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ba0a3dfc0a454959a06b57a3c732ea50 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ba0a3dfc0a454959a06b57a3c732ea50 2023-03-26 11:53:54,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ba0a3dfc0a454959a06b57a3c732ea50, entries=100, sequenceid=478, filesize=9.5 K 2023-03-26 11:53:54,294 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/c4c18af8af9d4e0b84b79188a10f4e3d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/c4c18af8af9d4e0b84b79188a10f4e3d 2023-03-26 11:53:54,299 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/c4c18af8af9d4e0b84b79188a10f4e3d, entries=100, sequenceid=478, filesize=9.5 K 2023-03-26 11:53:54,300 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51042bc7573c44eb24dbd242f395f3ff in 892ms, sequenceid=478, compaction requested=true 2023-03-26 11:53:54,300 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:54,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:54,300 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:53:54,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:54,300 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 11:53:54,301 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:53:54,301 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:53:54,302 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:54,302 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,302 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b25dab08206f4d13ac4dc992c08d9139, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9dbe29acff6b4500989a2ab43a42bf4b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=34.0 K 2023-03-26 11:53:54,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:53:54,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:54,302 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,302 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b25dab08206f4d13ac4dc992c08d9139, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679831631401 2023-03-26 11:53:54,302 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1183f985188c48149ff2a7a1f933e266, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/3531697cc60246ef8f99e040d11c62ec, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ba0a3dfc0a454959a06b57a3c732ea50] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=34.0 K 2023-03-26 11:53:54,303 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9dbe29acff6b4500989a2ab43a42bf4b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679831632603 2023-03-26 11:53:54,303 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1183f985188c48149ff2a7a1f933e266, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679831631401 2023-03-26 11:53:54,303 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ad8b39c65ff6449d9e0d1c5a6b5df0e7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679831632727 2023-03-26 11:53:54,304 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3531697cc60246ef8f99e040d11c62ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679831632603 2023-03-26 11:53:54,304 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ba0a3dfc0a454959a06b57a3c732ea50, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679831632727 2023-03-26 11:53:54,317 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#75 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:54,323 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#76 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:54,353 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/4bc99f1eeaf449d7be0bbf2a54d9bc9c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/4bc99f1eeaf449d7be0bbf2a54d9bc9c 2023-03-26 11:53:54,354 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/bd18df5a93a54d61ac96eab30ebc8c5b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/bd18df5a93a54d61ac96eab30ebc8c5b 2023-03-26 11:53:54,357 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:54,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:54,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:54,373 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into bd18df5a93a54d61ac96eab30ebc8c5b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:54,373 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:54,373 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=13, startTime=1679831634300; duration=0sec 2023-03-26 11:53:54,374 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:54,374 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:53:54,374 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into 4bc99f1eeaf449d7be0bbf2a54d9bc9c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:54,374 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:54,374 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=13, startTime=1679831634300; duration=0sec 2023-03-26 11:53:54,374 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:54,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:54,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:54,380 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:54,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:54,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:54,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:54,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:54,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:54,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:54,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:54,389 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:54,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:54,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:54,390 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 34864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:53:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:54,390 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:54,390 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,390 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/586bdfa32db9424897d148ad90cb761b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/a405458e159b458d98244fb511210a71, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/c4c18af8af9d4e0b84b79188a10f4e3d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=34.0 K 2023-03-26 11:53:54,391 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 586bdfa32db9424897d148ad90cb761b, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1679831631401 2023-03-26 11:53:54,391 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a405458e159b458d98244fb511210a71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1679831632603 2023-03-26 11:53:54,392 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c4c18af8af9d4e0b84b79188a10f4e3d, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679831632727 2023-03-26 11:53:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:54,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:54,402 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:54,409 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-26 11:53:54,409 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:54,409 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:54,409 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:54,409 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:54,409 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:54,409 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:54,425 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#78 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:54,436 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/054abdbd27434060b6adbe652c3c31fc 2023-03-26 11:53:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:54,592 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,601 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831694596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831694597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831694599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,705 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831694703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831694703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,706 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831694706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,841 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/8ab1f0d0e05e47b9ac4e4c36e250910a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8ab1f0d0e05e47b9ac4e4c36e250910a 2023-03-26 11:53:54,846 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into 8ab1f0d0e05e47b9ac4e4c36e250910a(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:54,846 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:54,846 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=13, startTime=1679831634300; duration=0sec 2023-03-26 11:53:54,847 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:54,858 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f06fabe294dc470690de4a46880e081b 2023-03-26 11:53:54,907 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831694907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831694907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:54,911 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:54,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831694909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831695210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,213 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831695211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831695216, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,270 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=517 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/cfdbf48c73284bbb8ccf78b748417119 2023-03-26 11:53:55,277 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/054abdbd27434060b6adbe652c3c31fc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/054abdbd27434060b6adbe652c3c31fc 2023-03-26 11:53:55,282 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/054abdbd27434060b6adbe652c3c31fc, entries=150, sequenceid=517, filesize=11.9 K 2023-03-26 11:53:55,283 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f06fabe294dc470690de4a46880e081b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f06fabe294dc470690de4a46880e081b 2023-03-26 11:53:55,288 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f06fabe294dc470690de4a46880e081b, entries=150, sequenceid=517, filesize=11.9 K 2023-03-26 11:53:55,289 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/cfdbf48c73284bbb8ccf78b748417119 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/cfdbf48c73284bbb8ccf78b748417119 2023-03-26 11:53:55,298 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/cfdbf48c73284bbb8ccf78b748417119, entries=150, sequenceid=517, filesize=11.9 K 2023-03-26 11:53:55,299 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 51042bc7573c44eb24dbd242f395f3ff in 894ms, sequenceid=517, compaction requested=false 2023-03-26 11:53:55,299 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:55,299 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:55,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:55,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:55,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:55,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:55,301 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:55,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:55,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:55,303 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:55,305 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b1eafd[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:55,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:55,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,306 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:55,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:55,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:55,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:55,309 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:55,309 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:55,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:55,311 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:55,311 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:55,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:55,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:55,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:55,312 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:55,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:55,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:55,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,318 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:55,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:55,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,319 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,319 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:55,419 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:55,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:55,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:55,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:55,424 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:55,424 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:55,424 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:55,424 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:55,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,426 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,426 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:55,427 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,427 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:55,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,427 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:55,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:55,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:55,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:55,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:55,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:55,440 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:55,440 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:55,440 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 11:53:55,446 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:55,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:55,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:55,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:55,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:55,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:55,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:55,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:55,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831635725, max timestamp=1679831635735Num uniques 859; 2023-03-26 11:53:55,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:55,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831635725, max timestamp=1679831635735Num uniques 859; 2023-03-26 11:53:55,737 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:55,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831635725, max timestamp=1679831635735Num uniques 859; 2023-03-26 11:53:55,738 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:53:55,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1679831635738, max timestamp=1679831635768Num uniques 1068; 2023-03-26 11:53:55,771 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=21428, suffix heap size=139040,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=468 2023-03-26 11:53:55,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1679831635738, max timestamp=1679831635768Num uniques 1068; 2023-03-26 11:53:55,773 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=21428, suffix heap size=139040,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=468 2023-03-26 11:53:55,775 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=468, cellSize=21428, totalHeapSize=48912, min timestamp=1679831635738, max timestamp=1679831635768Num uniques 1068; 2023-03-26 11:53:55,776 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=21428, suffix heap size=139040,new segment heap size=48912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=468 2023-03-26 11:53:55,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831695794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,796 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831695793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831695795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,871 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/ac3d8c9477804105b87d7699e9c78999 2023-03-26 11:53:55,886 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/c6d050ec01124e90a0c73187675679f4 2023-03-26 11:53:55,899 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=527 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/d87857a836704f2d96f5e1a80bb07633 2023-03-26 11:53:55,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831695898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831695898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,900 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:55,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831695899, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,904 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/ac3d8c9477804105b87d7699e9c78999 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ac3d8c9477804105b87d7699e9c78999 2023-03-26 11:53:55,909 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ac3d8c9477804105b87d7699e9c78999, entries=150, sequenceid=527, filesize=11.9 K 2023-03-26 11:53:55,910 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/c6d050ec01124e90a0c73187675679f4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c6d050ec01124e90a0c73187675679f4 2023-03-26 11:53:55,915 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c6d050ec01124e90a0c73187675679f4, entries=150, sequenceid=527, filesize=11.9 K 2023-03-26 11:53:55,916 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/d87857a836704f2d96f5e1a80bb07633 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d87857a836704f2d96f5e1a80bb07633 2023-03-26 11:53:55,922 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d87857a836704f2d96f5e1a80bb07633, entries=150, sequenceid=527, filesize=11.9 K 2023-03-26 11:53:55,922 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 51042bc7573c44eb24dbd242f395f3ff in 482ms, sequenceid=527, compaction requested=true 2023-03-26 11:53:55,922 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:55,923 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:55,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:55,926 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:55,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:55,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680cfd53[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:55,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:55,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,929 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:55,929 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:55,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:55,930 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:55,930 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,930 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,930 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:55,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:55,933 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:55,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:55,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:55,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:55,933 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:55,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:55,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:55,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:55,935 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,937 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:55,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:55,941 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:55,940 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:55,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:55,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:55,941 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:55,941 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:55,941 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:55,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,041 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:56,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:56,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:56,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:56,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:56,045 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:56,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:56,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:56,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:56,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,046 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:56,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:56,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:56,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:56,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,048 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:56,049 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:56,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:56,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:56,050 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:56,050 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:56,050 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:56,050 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=147.60 KB heapSize=395.86 KB 2023-03-26 11:53:56,050 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:56,050 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:56,051 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:56,051 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:56,051 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:56,051 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:56,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:56,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:56,118 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831696114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831696114, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831696119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831696220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831696221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831696222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831696424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831696425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831696427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,462 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/2159aafc87bf494bb21fad59d5f73b66 2023-03-26 11:53:56,494 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/632225da7ef540279f89e11377d1d227 2023-03-26 11:53:56,535 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=577 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/ec4e9d09f8d94cd8a8c382d29984a632 2023-03-26 11:53:56,557 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/2159aafc87bf494bb21fad59d5f73b66 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2159aafc87bf494bb21fad59d5f73b66 2023-03-26 11:53:56,564 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2159aafc87bf494bb21fad59d5f73b66, entries=150, sequenceid=577, filesize=11.9 K 2023-03-26 11:53:56,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/632225da7ef540279f89e11377d1d227 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/632225da7ef540279f89e11377d1d227 2023-03-26 11:53:56,573 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/632225da7ef540279f89e11377d1d227, entries=150, sequenceid=577, filesize=11.9 K 2023-03-26 11:53:56,574 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/ec4e9d09f8d94cd8a8c382d29984a632 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ec4e9d09f8d94cd8a8c382d29984a632 2023-03-26 11:53:56,585 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ec4e9d09f8d94cd8a8c382d29984a632, entries=150, sequenceid=577, filesize=11.9 K 2023-03-26 11:53:56,586 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~395.81 KB/405312, currentSize=40.25 KB/41220 for 51042bc7573c44eb24dbd242f395f3ff in 536ms, sequenceid=577, compaction requested=true 2023-03-26 11:53:56,586 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:56,586 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:56,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:56,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:56,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:56,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:56,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,589 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,590 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,591 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:56,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:56,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ed7e11[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:56,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:56,595 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,596 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:56,596 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:56,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:56,598 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:56,598 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:56,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:56,604 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:56,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:56,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:56,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:56,605 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:56,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,611 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,621 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:56,621 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:56,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:56,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:56,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:56,624 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:56,624 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:56,724 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:56,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:56,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:56,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:56,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:56,742 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:56,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:56,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:56,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:56,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:56,748 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:56,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,749 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,750 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:56,756 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:56,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:56,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:56,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:56,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:56,757 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:56,757 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:56,763 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:56,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:56,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:56,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:56,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:56,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:56,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:56,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:56,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:56,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:56,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@748973da[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:56,778 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:56,778 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:56,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:56,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:56,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:56,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:56,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:56,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:56,784 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:56,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:56,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:56,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:56,784 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:56,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:56,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,792 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:56,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:56,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:56,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:56,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:56,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:56,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,796 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:56,796 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:56,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:56,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:56,802 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/88c356a6d4fe4e41a7279aa6db33d5dc 2023-03-26 11:53:56,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831636785, max timestamp=1679831636802Num uniques 859; 2023-03-26 11:53:56,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:53:56,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831636785, max timestamp=1679831636802Num uniques 859; 2023-03-26 11:53:56,804 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:53:56,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831636785, max timestamp=1679831636802Num uniques 859; 2023-03-26 11:53:56,805 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:53:56,837 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831696830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831696831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831696833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,896 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:56,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:56,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:56,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:56,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:56,899 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:56,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:56,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:56,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:56,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,901 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:56,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:56,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:56,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:56,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:56,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:56,903 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:56,903 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:56,903 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:56,903 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:56,904 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:56,905 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,905 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,905 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,905 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:56,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:56,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d4699b4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:56,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:56,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,909 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:56,909 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:56,909 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:56,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:56,911 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:56,914 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:56,914 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:56,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:56,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:56,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:56,914 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:56,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:56,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:56,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,917 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:56,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:56,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:56,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:56,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:56,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:56,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:56,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:56,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:56,922 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:56,922 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:56,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:56,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:56,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:56,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,940 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831696939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831696939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:56,941 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:56,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831696939, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,025 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,027 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,028 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,028 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,028 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,028 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,029 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,029 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,029 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,029 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,031 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,031 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,031 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,031 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,033 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52af112a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,038 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,038 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,044 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,044 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,045 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,143 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831697141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831697143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,144 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831697143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,145 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,148 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,152 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,153 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,153 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,158 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dedf098[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,160 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,160 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,162 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,170 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,171 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,173 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,174 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,177 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,178 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,178 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,179 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:57,237 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/fa9bd8a8cac540068194114ca672ca38 2023-03-26 11:53:57,255 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=587 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/262320cef9ff4073801c693d4727f686 2023-03-26 11:53:57,270 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/88c356a6d4fe4e41a7279aa6db33d5dc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/88c356a6d4fe4e41a7279aa6db33d5dc 2023-03-26 11:53:57,276 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/88c356a6d4fe4e41a7279aa6db33d5dc, entries=200, sequenceid=587, filesize=14.3 K 2023-03-26 11:53:57,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/fa9bd8a8cac540068194114ca672ca38 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fa9bd8a8cac540068194114ca672ca38 2023-03-26 11:53:57,278 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,288 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,289 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,289 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fa9bd8a8cac540068194114ca672ca38, entries=150, sequenceid=587, filesize=11.9 K 2023-03-26 11:53:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,296 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/262320cef9ff4073801c693d4727f686 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/262320cef9ff4073801c693d4727f686 2023-03-26 11:53:57,296 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,296 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,296 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,297 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,301 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/262320cef9ff4073801c693d4727f686, entries=150, sequenceid=587, filesize=11.9 K 2023-03-26 11:53:57,301 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 51042bc7573c44eb24dbd242f395f3ff in 545ms, sequenceid=587, compaction requested=true 2023-03-26 11:53:57,302 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:57,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:53:57,302 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:53:57,302 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,304 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:53:57,304 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:53:57,304 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,304 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,304 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/4bc99f1eeaf449d7be0bbf2a54d9bc9c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f06fabe294dc470690de4a46880e081b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c6d050ec01124e90a0c73187675679f4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/632225da7ef540279f89e11377d1d227, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fa9bd8a8cac540068194114ca672ca38] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=60.3 K 2023-03-26 11:53:57,305 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64222 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:53:57,305 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:53:57,305 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,305 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/bd18df5a93a54d61ac96eab30ebc8c5b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/054abdbd27434060b6adbe652c3c31fc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ac3d8c9477804105b87d7699e9c78999, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2159aafc87bf494bb21fad59d5f73b66, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/88c356a6d4fe4e41a7279aa6db33d5dc] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=62.7 K 2023-03-26 11:53:57,305 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 4bc99f1eeaf449d7be0bbf2a54d9bc9c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679831632603 2023-03-26 11:53:57,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,306 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting bd18df5a93a54d61ac96eab30ebc8c5b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679831632603 2023-03-26 11:53:57,306 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f06fabe294dc470690de4a46880e081b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679831633461 2023-03-26 11:53:57,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,306 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 054abdbd27434060b6adbe652c3c31fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679831633461 2023-03-26 11:53:57,306 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c6d050ec01124e90a0c73187675679f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679831634593 2023-03-26 11:53:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,307 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ac3d8c9477804105b87d7699e9c78999, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679831634593 2023-03-26 11:53:57,307 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 632225da7ef540279f89e11377d1d227, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679831635788 2023-03-26 11:53:57,307 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2159aafc87bf494bb21fad59d5f73b66, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679831635788 2023-03-26 11:53:57,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,307 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fa9bd8a8cac540068194114ca672ca38, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1679831636108 2023-03-26 11:53:57,308 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 88c356a6d4fe4e41a7279aa6db33d5dc, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1679831636103 2023-03-26 11:53:57,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,308 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32c5455c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,308 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,308 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,309 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,311 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,319 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,319 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,323 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,323 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,323 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,325 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,327 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:57,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,332 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,332 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,338 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#90 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:57,350 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#91 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:57,406 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/b96690b3fd1d49e2a2130d9a5ad8b63f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b96690b3fd1d49e2a2130d9a5ad8b63f 2023-03-26 11:53:57,414 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into b96690b3fd1d49e2a2130d9a5ad8b63f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:57,414 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:57,415 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=11, startTime=1679831637302; duration=0sec 2023-03-26 11:53:57,415 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:53:57,415 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:53:57,420 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61782 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:53:57,421 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:53:57,421 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,421 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8ab1f0d0e05e47b9ac4e4c36e250910a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/cfdbf48c73284bbb8ccf78b748417119, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d87857a836704f2d96f5e1a80bb07633, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ec4e9d09f8d94cd8a8c382d29984a632, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/262320cef9ff4073801c693d4727f686] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=60.3 K 2023-03-26 11:53:57,423 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8ab1f0d0e05e47b9ac4e4c36e250910a, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1679831632603 2023-03-26 11:53:57,423 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting cfdbf48c73284bbb8ccf78b748417119, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=517, earliestPutTs=1679831633461 2023-03-26 11:53:57,424 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d87857a836704f2d96f5e1a80bb07633, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=527, earliestPutTs=1679831634593 2023-03-26 11:53:57,424 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ec4e9d09f8d94cd8a8c382d29984a632, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=577, earliestPutTs=1679831635788 2023-03-26 11:53:57,425 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 262320cef9ff4073801c693d4727f686, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1679831636108 2023-03-26 11:53:57,432 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,435 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,447 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/c28fcda6165f4e7ca215ea36d5730d7c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c28fcda6165f4e7ca215ea36d5730d7c 2023-03-26 11:53:57,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:57,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=147.60 KB heapSize=410.77 KB 2023-03-26 11:53:57,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:57,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:57,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:57,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:57,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:57,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:57,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,456 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,458 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,460 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,462 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#92 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:53:57,463 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into c28fcda6165f4e7ca215ea36d5730d7c(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:57,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:57,464 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=11, startTime=1679831637302; duration=0sec 2023-03-26 11:53:57,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:57,468 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,468 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,468 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,485 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831697477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831697484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,492 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,492 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,492 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,492 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,493 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,496 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831697487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,517 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b27f189[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,527 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/d96843c82e3c4de39be6f51f65f70079 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d96843c82e3c4de39be6f51f65f70079 2023-03-26 11:53:57,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,528 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,528 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,532 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,532 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,532 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,532 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,532 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,538 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into d96843c82e3c4de39be6f51f65f70079(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:53:57,538 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:57,538 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=11, startTime=1679831637302; duration=0sec 2023-03-26 11:53:57,539 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,539 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,542 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,542 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,586 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831697586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,590 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831697589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,600 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831697597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,642 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,646 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,648 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,649 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,649 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,650 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,650 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,650 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,651 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,653 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,653 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,653 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,653 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,655 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@712b9d86[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,655 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,655 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,658 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,659 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,662 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,662 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,664 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,670 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,670 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,674 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:57,770 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,774 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,775 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,776 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,776 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,777 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,777 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,777 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,777 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,777 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,778 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1320ca0b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,782 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,782 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,782 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,782 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,787 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,787 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,787 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,787 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,787 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,789 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 286 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831697788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,794 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,795 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,795 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831697793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,802 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:57,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831697802, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,895 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:57,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:57,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:57,898 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:57,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:57,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:57,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,900 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:57,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:57,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:57,902 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:57,902 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:57,902 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:57,902 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:57,902 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:57,902 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,902 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:57,903 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:57,903 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:57,903 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:57,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,907 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:57,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bd78296[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:57,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:57,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,910 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:57,911 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:57,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:57,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:57,918 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:57,918 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:57,918 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:57,918 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:57,918 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:57,918 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:57,919 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,919 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,919 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:57,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:57,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:57,930 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:57,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:57,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:57,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:57,934 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:57,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:57,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:57,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:57,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:57,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:57,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:57,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:57,934 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:57,934 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:57,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:57,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:57,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:57,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/a948719e8859406488b83a356d47d375 2023-03-26 11:53:57,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ae118c4717054034975d2c0922ac526e 2023-03-26 11:53:57,995 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=624 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/668c012162f34389a0510d42aae65702 2023-03-26 11:53:58,001 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/a948719e8859406488b83a356d47d375 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a948719e8859406488b83a356d47d375 2023-03-26 11:53:58,006 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a948719e8859406488b83a356d47d375, entries=150, sequenceid=624, filesize=11.9 K 2023-03-26 11:53:58,009 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ae118c4717054034975d2c0922ac526e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ae118c4717054034975d2c0922ac526e 2023-03-26 11:53:58,018 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ae118c4717054034975d2c0922ac526e, entries=150, sequenceid=624, filesize=11.9 K 2023-03-26 11:53:58,020 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/668c012162f34389a0510d42aae65702 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/668c012162f34389a0510d42aae65702 2023-03-26 11:53:58,029 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/668c012162f34389a0510d42aae65702, entries=150, sequenceid=624, filesize=11.9 K 2023-03-26 11:53:58,031 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~410.72 KB/420576, currentSize=46.96 KB/48090 for 51042bc7573c44eb24dbd242f395f3ff in 580ms, sequenceid=624, compaction requested=false 2023-03-26 11:53:58,031 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:58,034 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:58,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:58,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:58,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:58,038 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:58,038 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:58,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:58,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:58,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:58,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:58,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:58,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:58,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:58,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:58,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:58,041 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:58,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:58,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:58,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:58,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:58,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:58,044 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:58,044 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:58,045 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:53:58,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:58,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:58,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:58,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:58,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:58,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:53:58,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:53:58,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:53:58,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831638117, max timestamp=1679831638128Num uniques 859; 2023-03-26 11:53:58,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:58,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831638117, max timestamp=1679831638128Num uniques 859; 2023-03-26 11:53:58,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:53:58,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831638121, max timestamp=1679831638128Num uniques 859; 2023-03-26 11:53:58,132 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:53:58,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 288 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831698148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 285 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831698149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,151 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831698151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831698253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831698253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831698253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831698457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,460 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831698457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831698457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,482 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/dffea136ed1f4c5b8048184e9efda261 2023-03-26 11:53:58,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831698762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831698763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,766 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:58,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831698763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:58,897 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/9acf876d38cf40f8b537a74b027c067e 2023-03-26 11:53:59,270 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831699269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,271 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:59,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831699269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:53:59,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831699271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,334 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=636 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/d5dd756327b64e80898f17d01d1197ba 2023-03-26 11:53:59,340 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/dffea136ed1f4c5b8048184e9efda261 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/dffea136ed1f4c5b8048184e9efda261 2023-03-26 11:53:59,346 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/dffea136ed1f4c5b8048184e9efda261, entries=150, sequenceid=636, filesize=11.9 K 2023-03-26 11:53:59,347 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/9acf876d38cf40f8b537a74b027c067e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/9acf876d38cf40f8b537a74b027c067e 2023-03-26 11:53:59,353 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/9acf876d38cf40f8b537a74b027c067e, entries=150, sequenceid=636, filesize=11.9 K 2023-03-26 11:53:59,354 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/d5dd756327b64e80898f17d01d1197ba as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d5dd756327b64e80898f17d01d1197ba 2023-03-26 11:53:59,359 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d5dd756327b64e80898f17d01d1197ba, entries=150, sequenceid=636, filesize=11.9 K 2023-03-26 11:53:59,360 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=136.02 KB/139285 for 51042bc7573c44eb24dbd242f395f3ff in 1316ms, sequenceid=636, compaction requested=true 2023-03-26 11:53:59,360 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:59,360 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:59,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:59,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:59,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:59,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,362 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:59,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:59,362 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,363 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:59,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1891dc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:59,364 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:59,364 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,365 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:59,365 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,365 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:59,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:59,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:59,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:59,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:59,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:59,374 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:59,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:59,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:59,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:59,374 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:59,376 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,376 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,376 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:59,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:59,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:59,378 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:59,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:59,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:59,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:59,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:59,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:59,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:53:59,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,381 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:59,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:59,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:59,381 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:59,481 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:53:59,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:53:59,483 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:53:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:53:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:53:59,484 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:53:59,484 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:59,484 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:53:59,484 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:53:59,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,487 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:59,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:59,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:53:59,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,488 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:53:59,488 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:53:59,488 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:53:59,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:53:59,488 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:53:59,489 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=136.02 KB heapSize=384.46 KB 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:53:59,489 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:53:59,899 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/5470dab207c742589bbadd3645a1699b 2023-03-26 11:53:59,914 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f3992efe1bc04723bcf9c61c1d44bd25 2023-03-26 11:53:59,931 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.09 KB at sequenceid=672 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/8fc3641c1cdb4552b911061a0d342453 2023-03-26 11:53:59,942 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/5470dab207c742589bbadd3645a1699b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5470dab207c742589bbadd3645a1699b 2023-03-26 11:53:59,947 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5470dab207c742589bbadd3645a1699b, entries=150, sequenceid=672, filesize=11.9 K 2023-03-26 11:53:59,949 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/f3992efe1bc04723bcf9c61c1d44bd25 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3992efe1bc04723bcf9c61c1d44bd25 2023-03-26 11:53:59,954 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3992efe1bc04723bcf9c61c1d44bd25, entries=150, sequenceid=672, filesize=11.9 K 2023-03-26 11:53:59,955 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/8fc3641c1cdb4552b911061a0d342453 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8fc3641c1cdb4552b911061a0d342453 2023-03-26 11:53:59,960 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8fc3641c1cdb4552b911061a0d342453, entries=150, sequenceid=672, filesize=11.9 K 2023-03-26 11:53:59,961 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~136.02 KB/139285, heapSize ~384.41 KB/393640, currentSize=0 B/0 for 51042bc7573c44eb24dbd242f395f3ff in 472ms, sequenceid=672, compaction requested=true 2023-03-26 11:53:59,961 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:53:59,961 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:53:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:53:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:53:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:53:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:53:59,963 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:53:59,966 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42f2f614[Count = 0] remaining members to acquire global barrier 2023-03-26 11:53:59,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:53:59,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,967 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:53:59,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:53:59,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:53:59,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:59,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:53:59,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:59,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:53:59,974 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:53:59,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:53:59,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:53:59,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:53:59,975 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:53:59,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:53:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:53:59,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:53:59,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:53:59,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:53:59,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:53:59,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:53:59,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:53:59,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:53:59,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:53:59,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:53:59,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,983 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:53:59,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:53:59,983 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:53:59,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:53:59,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:53:59,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,084 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,087 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,087 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,087 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,087 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,089 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,089 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,090 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:00,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,091 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,092 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,092 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,092 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:00,092 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,094 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,094 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57971cd3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,099 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,101 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,103 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,103 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,103 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,107 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,109 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,112 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,112 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,112 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,212 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,215 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,216 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,218 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,218 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,221 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,221 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,221 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:00,221 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,222 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60e3e785[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,225 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,226 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,226 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,227 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,229 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,229 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,229 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,229 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,229 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,232 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,232 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,232 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,245 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,245 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,246 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,246 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:00,280 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:00,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:54:00,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:00,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:54:00,281 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:00,281 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:54:00,282 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:00,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831640301, max timestamp=1679831640310Num uniques 859; 2023-03-26 11:54:00,312 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:00,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=109, cellSize=4985, totalHeapSize=11576, min timestamp=1679831640306, max timestamp=1679831640310Num uniques 859; 2023-03-26 11:54:00,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=4985, suffix heap size=117064,new segment heap size=11576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=109 2023-03-26 11:54:00,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=109, cellSize=4985, totalHeapSize=11576, min timestamp=1679831640306, max timestamp=1679831640310Num uniques 859; 2023-03-26 11:54:00,313 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=4985, suffix heap size=117064,new segment heap size=11576 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=109 2023-03-26 11:54:00,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831700332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,337 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831700332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831700336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,346 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,348 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:00,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,351 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,351 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,351 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:00,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,353 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,353 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bde1c34[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,359 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,361 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,361 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,361 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,361 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,361 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,363 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,364 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:00,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,367 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,367 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,367 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831700438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,441 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831700438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,446 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831700443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,468 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,470 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,471 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,474 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,475 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,475 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,475 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:00,475 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:00,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,478 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,480 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e49010d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,480 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,481 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,481 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,481 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,481 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,482 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,485 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,485 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,485 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,488 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,488 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,488 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,490 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,493 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,493 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,493 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:00,593 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,595 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,597 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:00,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,599 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,599 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,599 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:00,599 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,602 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71d7e793[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,602 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,605 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,605 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,609 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,609 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,612 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,614 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,620 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,621 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,621 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,621 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,621 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,621 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,621 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,621 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831700642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,645 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831700642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,653 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831700648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/8afb7ce4d66f4bd0957ce49961ddd5bd 2023-03-26 11:54:00,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/1765ed9ffc3245129ead02a8b9a3347e 2023-03-26 11:54:00,721 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,725 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,726 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,726 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,735 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,736 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,736 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,737 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=685 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/0791125b17b043e1bd3bc5504116335d 2023-03-26 11:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,737 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,737 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:00,738 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,738 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,738 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,738 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:00,738 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,739 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,741 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,741 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,742 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,742 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,743 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/8afb7ce4d66f4bd0957ce49961ddd5bd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8afb7ce4d66f4bd0957ce49961ddd5bd 2023-03-26 11:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ef7f9e3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,745 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,745 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,745 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,749 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,749 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,751 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8afb7ce4d66f4bd0957ce49961ddd5bd, entries=300, sequenceid=685, filesize=19.0 K 2023-03-26 11:54:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,758 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/1765ed9ffc3245129ead02a8b9a3347e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1765ed9ffc3245129ead02a8b9a3347e 2023-03-26 11:54:00,760 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,760 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,760 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,760 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,763 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1765ed9ffc3245129ead02a8b9a3347e, entries=150, sequenceid=685, filesize=11.9 K 2023-03-26 11:54:00,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/0791125b17b043e1bd3bc5504116335d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/0791125b17b043e1bd3bc5504116335d 2023-03-26 11:54:00,766 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,766 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,766 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,766 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,766 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,767 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,766 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,767 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,771 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/0791125b17b043e1bd3bc5504116335d, entries=150, sequenceid=685, filesize=11.9 K 2023-03-26 11:54:00,772 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 51042bc7573c44eb24dbd242f395f3ff in 492ms, sequenceid=685, compaction requested=true 2023-03-26 11:54:00,772 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:00,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:00,772 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:54:00,772 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:54:00,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:00,772 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:00,774 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 69272 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:54:00,774 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:54:00,774 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61952 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:54:00,774 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,774 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:54:00,774 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b96690b3fd1d49e2a2130d9a5ad8b63f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a948719e8859406488b83a356d47d375, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/dffea136ed1f4c5b8048184e9efda261, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5470dab207c742589bbadd3645a1699b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8afb7ce4d66f4bd0957ce49961ddd5bd] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=67.6 K 2023-03-26 11:54:00,774 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,775 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c28fcda6165f4e7ca215ea36d5730d7c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ae118c4717054034975d2c0922ac526e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/9acf876d38cf40f8b537a74b027c067e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3992efe1bc04723bcf9c61c1d44bd25, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1765ed9ffc3245129ead02a8b9a3347e] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=60.5 K 2023-03-26 11:54:00,775 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b96690b3fd1d49e2a2130d9a5ad8b63f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1679831636108 2023-03-26 11:54:00,775 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c28fcda6165f4e7ca215ea36d5730d7c, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1679831636108 2023-03-26 11:54:00,775 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a948719e8859406488b83a356d47d375, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1679831636817 2023-03-26 11:54:00,776 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ae118c4717054034975d2c0922ac526e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1679831636817 2023-03-26 11:54:00,776 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting dffea136ed1f4c5b8048184e9efda261, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1679831637474 2023-03-26 11:54:00,776 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9acf876d38cf40f8b537a74b027c067e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1679831637474 2023-03-26 11:54:00,776 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5470dab207c742589bbadd3645a1699b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679831638142 2023-03-26 11:54:00,776 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f3992efe1bc04723bcf9c61c1d44bd25, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679831638142 2023-03-26 11:54:00,777 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8afb7ce4d66f4bd0957ce49961ddd5bd, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679831640277 2023-03-26 11:54:00,777 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1765ed9ffc3245129ead02a8b9a3347e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679831640277 2023-03-26 11:54:00,789 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#105 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:00,789 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#106 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:00,867 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:00,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:00,871 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:00,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:00,873 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:00,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:00,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:00,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:00,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:00,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:00,878 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:00,878 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:00,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:00,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:00,879 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=125.23 KB heapSize=359.98 KB 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:54:00,879 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:00,906 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/d5eafcb495b646f582c9bcb982a70ae9 2023-03-26 11:54:00,926 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/8fb8fbc234344ff385a901236cb42a02 2023-03-26 11:54:00,940 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=721 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/ad73b30f423945bb88336d0c199241fd 2023-03-26 11:54:00,946 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/d5eafcb495b646f582c9bcb982a70ae9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d5eafcb495b646f582c9bcb982a70ae9 2023-03-26 11:54:00,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:00,949 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:00,952 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d5eafcb495b646f582c9bcb982a70ae9, entries=150, sequenceid=721, filesize=11.9 K 2023-03-26 11:54:00,954 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/8fb8fbc234344ff385a901236cb42a02 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/8fb8fbc234344ff385a901236cb42a02 2023-03-26 11:54:00,961 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/8fb8fbc234344ff385a901236cb42a02, entries=150, sequenceid=721, filesize=11.9 K 2023-03-26 11:54:00,962 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/ad73b30f423945bb88336d0c199241fd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ad73b30f423945bb88336d0c199241fd 2023-03-26 11:54:00,966 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ad73b30f423945bb88336d0c199241fd, entries=150, sequenceid=721, filesize=11.9 K 2023-03-26 11:54:00,967 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~359.94 KB/368576, currentSize=53.67 KB/54960 for 51042bc7573c44eb24dbd242f395f3ff in 88ms, sequenceid=721, compaction requested=true 2023-03-26 11:54:00,967 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:00,967 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:00,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:00,969 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:54:00,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:00,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:54:00,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:00,970 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:54:00,970 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:00,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35736bb5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:00,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:00,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,976 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:00,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:00,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:00,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:00,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:00,982 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:00,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:00,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:00,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:00,982 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:00,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:00,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:00,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:00,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:00,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:00,989 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:00,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:00,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:00,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:00,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:00,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:00,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:00,989 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:00,989 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:00,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:00,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:00,995 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831700992, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 327 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831700993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:00,996 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:00,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 340 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831700994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,089 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:01,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:01,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:01,093 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:01,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:01,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:01,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:01,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:01,096 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:01,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:01,100 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831701097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 324 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831701097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 342 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831701097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:01,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:01,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:01,108 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,108 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:01,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:01,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:01,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:01,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:01,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63e4457f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:01,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:01,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:01,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:01,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:01,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,116 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,117 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:01,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:01,119 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:01,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:01,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:01,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:01,119 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:01,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:01,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:01,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,123 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:01,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:01,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:01,127 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:01,127 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:01,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:01,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:01,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:01,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:01,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:01,212 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/efda1a71102943898b87555feb0dfbd7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/efda1a71102943898b87555feb0dfbd7 2023-03-26 11:54:01,216 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/620765007cc642b9b84e4cb466fd20a8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/620765007cc642b9b84e4cb466fd20a8 2023-03-26 11:54:01,222 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into 620765007cc642b9b84e4cb466fd20a8(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:01,222 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:01,222 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=11, startTime=1679831640772; duration=0sec 2023-03-26 11:54:01,222 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:01,222 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:54:01,223 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into efda1a71102943898b87555feb0dfbd7(size=13.1 K), total size for store is 25.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:01,223 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:01,223 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=11, startTime=1679831640772; duration=0sec 2023-03-26 11:54:01,223 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:01,225 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74138 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:54:01,225 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/C is initiating minor compaction (all files) 2023-03-26 11:54:01,225 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/C in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,226 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d96843c82e3c4de39be6f51f65f70079, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/668c012162f34389a0510d42aae65702, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d5dd756327b64e80898f17d01d1197ba, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8fc3641c1cdb4552b911061a0d342453, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/0791125b17b043e1bd3bc5504116335d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ad73b30f423945bb88336d0c199241fd] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=72.4 K 2023-03-26 11:54:01,226 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d96843c82e3c4de39be6f51f65f70079, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=587, earliestPutTs=1679831636108 2023-03-26 11:54:01,226 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 668c012162f34389a0510d42aae65702, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=624, earliestPutTs=1679831636817 2023-03-26 11:54:01,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d5dd756327b64e80898f17d01d1197ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=636, earliestPutTs=1679831637474 2023-03-26 11:54:01,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8fc3641c1cdb4552b911061a0d342453, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=672, earliestPutTs=1679831638142 2023-03-26 11:54:01,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0791125b17b043e1bd3bc5504116335d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679831640277 2023-03-26 11:54:01,228 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:01,228 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ad73b30f423945bb88336d0c199241fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1679831640330 2023-03-26 11:54:01,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:01,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:01,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:01,232 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:01,232 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:01,232 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:01,232 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:01,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,235 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:01,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:01,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:01,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:01,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:01,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:01,248 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:01,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:01,250 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:01,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,251 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#C#compaction#111 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:01,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2dc1cdef[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:01,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:01,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,255 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:01,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:01,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:01,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:01,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:01,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:01,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:01,260 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:01,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:01,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:01,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:01,261 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:01,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:01,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:01,263 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,265 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,267 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:01,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:01,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:01,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,268 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:01,268 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:01,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:01,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:01,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:01,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 344 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831701302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,305 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831701303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831701304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,368 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:01,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:01,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:01,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:01,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:01,373 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:01,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:01,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:01,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:01,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,375 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:01,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:01,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:01,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,376 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:01,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:01,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:01,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:01,377 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,377 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. started... 2023-03-26 11:54:01,377 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. as already flushing 2023-03-26 11:54:01,377 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:01,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:01,379 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,379 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/152c204810184be889e4090847aa9aeb 2023-03-26 11:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:01,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:01,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cc81155[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:01,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:01,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,386 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:01,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:01,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:01,389 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:01,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:01,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:01,392 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:01,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:01,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:01,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:01,392 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:01,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:01,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:01,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:01,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:01,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:01,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:01,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:01,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:01,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:01,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:01,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:01,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:01,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:01,399 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:01,399 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:01,435 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4b02eeb3 to 127.0.0.1:53955 2023-03-26 11:54:01,435 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:01,436 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x04fb2b82 to 127.0.0.1:53955 2023-03-26 11:54:01,436 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6129209e to 127.0.0.1:53955 2023-03-26 11:54:01,436 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:01,436 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1818ad0a to 127.0.0.1:53955 2023-03-26 11:54:01,436 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:01,436 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:01,500 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:01,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:01,607 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 346 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831701607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,609 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831701608, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,610 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:01,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831701610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:01,663 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/f84b596c03874fbf86230e6a49873a84 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/f84b596c03874fbf86230e6a49873a84 2023-03-26 11:54:01,668 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/C of 51042bc7573c44eb24dbd242f395f3ff into f84b596c03874fbf86230e6a49873a84(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:01,668 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:01,668 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/C, priority=10, startTime=1679831640772; duration=0sec 2023-03-26 11:54:01,668 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:01,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ce07157778f841ac977974d756209eb9 2023-03-26 11:54:02,109 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:02,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 348 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36284 deadline: 1679831702109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:02,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:02,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:02,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36258 deadline: 1679831702112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:02,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:36312 deadline: 1679831702112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:02,219 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=734 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/dc4cc8c7439443cf81b1be6aa7075d2e 2023-03-26 11:54:02,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/152c204810184be889e4090847aa9aeb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/152c204810184be889e4090847aa9aeb 2023-03-26 11:54:02,231 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/152c204810184be889e4090847aa9aeb, entries=350, sequenceid=734, filesize=21.4 K 2023-03-26 11:54:02,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ce07157778f841ac977974d756209eb9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ce07157778f841ac977974d756209eb9 2023-03-26 11:54:02,237 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ce07157778f841ac977974d756209eb9, entries=150, sequenceid=734, filesize=11.9 K 2023-03-26 11:54:02,238 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/dc4cc8c7439443cf81b1be6aa7075d2e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/dc4cc8c7439443cf81b1be6aa7075d2e 2023-03-26 11:54:02,244 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/dc4cc8c7439443cf81b1be6aa7075d2e, entries=150, sequenceid=734, filesize=11.9 K 2023-03-26 11:54:02,245 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 51042bc7573c44eb24dbd242f395f3ff in 1276ms, sequenceid=734, compaction requested=true 2023-03-26 11:54:02,245 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:02,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:02,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:02,245 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:02,245 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:02,245 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:02,248 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:02,248 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/B is initiating minor compaction (all files) 2023-03-26 11:54:02,248 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/B in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:02,248 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/efda1a71102943898b87555feb0dfbd7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/8fb8fbc234344ff385a901236cb42a02, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ce07157778f841ac977974d756209eb9] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=36.9 K 2023-03-26 11:54:02,249 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 47514 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:02,249 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 51042bc7573c44eb24dbd242f395f3ff/A is initiating minor compaction (all files) 2023-03-26 11:54:02,249 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting efda1a71102943898b87555feb0dfbd7, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679831640277 2023-03-26 11:54:02,249 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 51042bc7573c44eb24dbd242f395f3ff/A in TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:02,249 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/620765007cc642b9b84e4cb466fd20a8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d5eafcb495b646f582c9bcb982a70ae9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/152c204810184be889e4090847aa9aeb] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp, totalSize=46.4 K 2023-03-26 11:54:02,249 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8fb8fbc234344ff385a901236cb42a02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1679831640330 2023-03-26 11:54:02,250 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 620765007cc642b9b84e4cb466fd20a8, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=685, earliestPutTs=1679831640277 2023-03-26 11:54:02,250 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ce07157778f841ac977974d756209eb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1679831640959 2023-03-26 11:54:02,251 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d5eafcb495b646f582c9bcb982a70ae9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=721, earliestPutTs=1679831640330 2023-03-26 11:54:02,251 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 152c204810184be889e4090847aa9aeb, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=734, earliestPutTs=1679831640949 2023-03-26 11:54:02,267 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#B#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:02,269 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 51042bc7573c44eb24dbd242f395f3ff#A#compaction#115 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:02,281 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/9c3e091327c644dfae9262d32a6245fd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9c3e091327c644dfae9262d32a6245fd 2023-03-26 11:54:02,286 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/A of 51042bc7573c44eb24dbd242f395f3ff into 9c3e091327c644dfae9262d32a6245fd(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:02,286 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:02,286 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/A, priority=13, startTime=1679831642245; duration=0sec 2023-03-26 11:54:02,286 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:02,286 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 11:54:02,287 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 11:54:02,287 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 11:54:02,287 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. because compaction request was cancelled 2023-03-26 11:54:02,681 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/66c6f1610c1b441d892579a2b811c9bd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/66c6f1610c1b441d892579a2b811c9bd 2023-03-26 11:54:02,687 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 51042bc7573c44eb24dbd242f395f3ff/B of 51042bc7573c44eb24dbd242f395f3ff into 66c6f1610c1b441d892579a2b811c9bd(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:02,687 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:02,687 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff., storeName=51042bc7573c44eb24dbd242f395f3ff/B, priority=13, startTime=1679831642245; duration=0sec 2023-03-26 11:54:02,687 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:03,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:03,114 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:54:03,114 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:54:03,114 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0e16d41b to 127.0.0.1:53955 2023-03-26 11:54:03,114 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:03,114 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:03,114 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:54:03,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:03,115 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:54:03,115 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:03,120 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e728705 to 127.0.0.1:53955 2023-03-26 11:54:03,120 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:03,122 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1b886c8f to 127.0.0.1:53955 2023-03-26 11:54:03,122 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:03,179 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x77648431 to 127.0.0.1:53955 2023-03-26 11:54:03,179 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:03,198 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x09a575b8 to 127.0.0.1:53955 2023-03-26 11:54:03,198 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:03,198 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 182 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 200 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 35 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 185 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 37 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4134 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4110 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1839 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5514 rows 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1833 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5499 rows 2023-03-26 11:54:03,199 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:54:03,199 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2678d495 to 127.0.0.1:53955 2023-03-26 11:54:03,199 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:03,205 INFO [Listener at localhost/46409] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 11:54:03,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 11:54:03,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:03,225 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831643225"}]},"ts":"1679831643225"} 2023-03-26 11:54:03,226 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 11:54:03,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:03,229 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 11:54:03,231 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, UNASSIGN}] 2023-03-26 11:54:03,233 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, UNASSIGN 2023-03-26 11:54:03,234 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=51042bc7573c44eb24dbd242f395f3ff, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:03,234 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831643234"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831643234"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831643234"}]},"ts":"1679831643234"} 2023-03-26 11:54:03,236 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:03,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:03,395 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:03,395 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 51042bc7573c44eb24dbd242f395f3ff, disabling compactions & flushes 2023-03-26 11:54:03,396 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:03,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:03,538 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/4111df2a6db74c82b09abdb91f14ba60 2023-03-26 11:54:03,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:03,952 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/d3d589a17f334565b15079e89c606aa6 2023-03-26 11:54:04,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:04,363 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=759 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/8316ad2923484b28a5829b034b18dfc4 2023-03-26 11:54:04,368 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/4111df2a6db74c82b09abdb91f14ba60 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/4111df2a6db74c82b09abdb91f14ba60 2023-03-26 11:54:04,372 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/4111df2a6db74c82b09abdb91f14ba60, entries=150, sequenceid=759, filesize=11.9 K 2023-03-26 11:54:04,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/d3d589a17f334565b15079e89c606aa6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d3d589a17f334565b15079e89c606aa6 2023-03-26 11:54:04,377 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d3d589a17f334565b15079e89c606aa6, entries=150, sequenceid=759, filesize=11.9 K 2023-03-26 11:54:04,378 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/8316ad2923484b28a5829b034b18dfc4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8316ad2923484b28a5829b034b18dfc4 2023-03-26 11:54:04,382 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8316ad2923484b28a5829b034b18dfc4, entries=150, sequenceid=759, filesize=11.9 K 2023-03-26 11:54:04,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=26.84 KB/27480 for 51042bc7573c44eb24dbd242f395f3ff in 1269ms, sequenceid=759, compaction requested=true 2023-03-26 11:54:04,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:04,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:04,383 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. because compaction request was cancelled 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:04,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. after waiting 0 ms 2023-03-26 11:54:04,383 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. because compaction request was cancelled 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:04,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:04,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:04,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 51042bc7573c44eb24dbd242f395f3ff 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 11:54:04,383 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. because compaction request was cancelled 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=A 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=B 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:04,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 51042bc7573c44eb24dbd242f395f3ff, store=C 2023-03-26 11:54:04,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:04,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/5058a6268c8f4163bbac403f14338036 2023-03-26 11:54:05,201 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ac9444eaab174df98bd56bf96ca48f21 2023-03-26 11:54:05,331 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:05,611 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/19655494548145d5a6ab179a638d4291 2023-03-26 11:54:05,617 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/A/5058a6268c8f4163bbac403f14338036 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5058a6268c8f4163bbac403f14338036 2023-03-26 11:54:05,621 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5058a6268c8f4163bbac403f14338036, entries=150, sequenceid=766, filesize=11.9 K 2023-03-26 11:54:05,621 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/B/ac9444eaab174df98bd56bf96ca48f21 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ac9444eaab174df98bd56bf96ca48f21 2023-03-26 11:54:05,626 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ac9444eaab174df98bd56bf96ca48f21, entries=150, sequenceid=766, filesize=11.9 K 2023-03-26 11:54:05,626 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/.tmp/C/19655494548145d5a6ab179a638d4291 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/19655494548145d5a6ab179a638d4291 2023-03-26 11:54:05,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/19655494548145d5a6ab179a638d4291, entries=150, sequenceid=766, filesize=11.9 K 2023-03-26 11:54:05,631 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 51042bc7573c44eb24dbd242f395f3ff in 1248ms, sequenceid=766, compaction requested=true 2023-03-26 11:54:05,654 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a5c2c8c07421425b831d8b81a0675b65, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/1d8ee94d7db845509dbaac89bc9ff13a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2acc41ca7c8f424796482574547cf864, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9d3bfdae65184c01839e86b7d82205d9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/80e9faa422544ba6bf505d6a596b4b33, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/f92d1e1bc38145489d17a2760ac6d4cd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5004121d7e7c48c79dc01d8346371303, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/fb7eaed762374bb293949879721e7d89, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d08bb0da4c9c409eacdf7220a1b99775, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9bc80a05d11f4816a625999112fb1c96, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d117cd150e89412d80063cbf0ae926da, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/3c1e143cfaa24b8781c3cbd8370b2fae, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/67f6c230d10f48fc9f34b23fe14eab6a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2a4074cc920c4b199c68d013e32edbc0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a1b448ec12114067ac3036bbbe430d75, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/add24bcbf53d46f99f1ebf20172bdd3c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b84472cc8d314af5855dc9e84e59d79a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b0e4ded7b79f44c4bcca631589e1b65a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/df879f42fef04e7ba28d9eb8ece9cac8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8da865cfb8ac4f8b81041a110ead3bc1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/422126d619504ebeb7dd7ea13226e03a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b25dab08206f4d13ac4dc992c08d9139, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/740fae4105834d729ddedd22e98ced74, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9dbe29acff6b4500989a2ab43a42bf4b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/bd18df5a93a54d61ac96eab30ebc8c5b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/054abdbd27434060b6adbe652c3c31fc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ac3d8c9477804105b87d7699e9c78999, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2159aafc87bf494bb21fad59d5f73b66, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/88c356a6d4fe4e41a7279aa6db33d5dc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b96690b3fd1d49e2a2130d9a5ad8b63f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a948719e8859406488b83a356d47d375, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/dffea136ed1f4c5b8048184e9efda261, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5470dab207c742589bbadd3645a1699b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8afb7ce4d66f4bd0957ce49961ddd5bd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/620765007cc642b9b84e4cb466fd20a8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d5eafcb495b646f582c9bcb982a70ae9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/152c204810184be889e4090847aa9aeb] to archive 2023-03-26 11:54:05,655 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:54:05,661 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a5c2c8c07421425b831d8b81a0675b65 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a5c2c8c07421425b831d8b81a0675b65 2023-03-26 11:54:05,662 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/1d8ee94d7db845509dbaac89bc9ff13a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/1d8ee94d7db845509dbaac89bc9ff13a 2023-03-26 11:54:05,663 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2acc41ca7c8f424796482574547cf864 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2acc41ca7c8f424796482574547cf864 2023-03-26 11:54:05,664 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9d3bfdae65184c01839e86b7d82205d9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9d3bfdae65184c01839e86b7d82205d9 2023-03-26 11:54:05,666 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/80e9faa422544ba6bf505d6a596b4b33 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/80e9faa422544ba6bf505d6a596b4b33 2023-03-26 11:54:05,667 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/f92d1e1bc38145489d17a2760ac6d4cd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/f92d1e1bc38145489d17a2760ac6d4cd 2023-03-26 11:54:05,668 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5004121d7e7c48c79dc01d8346371303 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5004121d7e7c48c79dc01d8346371303 2023-03-26 11:54:05,669 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/fb7eaed762374bb293949879721e7d89 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/fb7eaed762374bb293949879721e7d89 2023-03-26 11:54:05,670 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d08bb0da4c9c409eacdf7220a1b99775 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d08bb0da4c9c409eacdf7220a1b99775 2023-03-26 11:54:05,671 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9bc80a05d11f4816a625999112fb1c96 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9bc80a05d11f4816a625999112fb1c96 2023-03-26 11:54:05,672 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d117cd150e89412d80063cbf0ae926da to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d117cd150e89412d80063cbf0ae926da 2023-03-26 11:54:05,673 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/3c1e143cfaa24b8781c3cbd8370b2fae to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/3c1e143cfaa24b8781c3cbd8370b2fae 2023-03-26 11:54:05,674 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/67f6c230d10f48fc9f34b23fe14eab6a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/67f6c230d10f48fc9f34b23fe14eab6a 2023-03-26 11:54:05,675 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2a4074cc920c4b199c68d013e32edbc0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2a4074cc920c4b199c68d013e32edbc0 2023-03-26 11:54:05,676 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a1b448ec12114067ac3036bbbe430d75 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a1b448ec12114067ac3036bbbe430d75 2023-03-26 11:54:05,678 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/add24bcbf53d46f99f1ebf20172bdd3c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/add24bcbf53d46f99f1ebf20172bdd3c 2023-03-26 11:54:05,679 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b84472cc8d314af5855dc9e84e59d79a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b84472cc8d314af5855dc9e84e59d79a 2023-03-26 11:54:05,680 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b0e4ded7b79f44c4bcca631589e1b65a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b0e4ded7b79f44c4bcca631589e1b65a 2023-03-26 11:54:05,681 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/df879f42fef04e7ba28d9eb8ece9cac8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/df879f42fef04e7ba28d9eb8ece9cac8 2023-03-26 11:54:05,682 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8da865cfb8ac4f8b81041a110ead3bc1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8da865cfb8ac4f8b81041a110ead3bc1 2023-03-26 11:54:05,683 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/422126d619504ebeb7dd7ea13226e03a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/422126d619504ebeb7dd7ea13226e03a 2023-03-26 11:54:05,684 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b25dab08206f4d13ac4dc992c08d9139 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b25dab08206f4d13ac4dc992c08d9139 2023-03-26 11:54:05,685 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/740fae4105834d729ddedd22e98ced74 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/740fae4105834d729ddedd22e98ced74 2023-03-26 11:54:05,686 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9dbe29acff6b4500989a2ab43a42bf4b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9dbe29acff6b4500989a2ab43a42bf4b 2023-03-26 11:54:05,687 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/bd18df5a93a54d61ac96eab30ebc8c5b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/bd18df5a93a54d61ac96eab30ebc8c5b 2023-03-26 11:54:05,688 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ad8b39c65ff6449d9e0d1c5a6b5df0e7 2023-03-26 11:54:05,689 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/054abdbd27434060b6adbe652c3c31fc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/054abdbd27434060b6adbe652c3c31fc 2023-03-26 11:54:05,690 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ac3d8c9477804105b87d7699e9c78999 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/ac3d8c9477804105b87d7699e9c78999 2023-03-26 11:54:05,691 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2159aafc87bf494bb21fad59d5f73b66 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/2159aafc87bf494bb21fad59d5f73b66 2023-03-26 11:54:05,692 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/88c356a6d4fe4e41a7279aa6db33d5dc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/88c356a6d4fe4e41a7279aa6db33d5dc 2023-03-26 11:54:05,693 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b96690b3fd1d49e2a2130d9a5ad8b63f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/b96690b3fd1d49e2a2130d9a5ad8b63f 2023-03-26 11:54:05,695 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a948719e8859406488b83a356d47d375 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/a948719e8859406488b83a356d47d375 2023-03-26 11:54:05,696 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/dffea136ed1f4c5b8048184e9efda261 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/dffea136ed1f4c5b8048184e9efda261 2023-03-26 11:54:05,697 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5470dab207c742589bbadd3645a1699b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5470dab207c742589bbadd3645a1699b 2023-03-26 11:54:05,699 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8afb7ce4d66f4bd0957ce49961ddd5bd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/8afb7ce4d66f4bd0957ce49961ddd5bd 2023-03-26 11:54:05,700 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/620765007cc642b9b84e4cb466fd20a8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/620765007cc642b9b84e4cb466fd20a8 2023-03-26 11:54:05,701 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d5eafcb495b646f582c9bcb982a70ae9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/d5eafcb495b646f582c9bcb982a70ae9 2023-03-26 11:54:05,702 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/152c204810184be889e4090847aa9aeb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/152c204810184be889e4090847aa9aeb 2023-03-26 11:54:05,727 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/eef069460f334e258c4565a5c28dcfb0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3059401bf68407887c87d879db9e3bd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/819b69e5e9e54bc7a3bdbd56ba555c27, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fc84b9a174124525962bc7c430fad71e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/dad4d1c050db437189e8fc7311347412, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/59d13b30ce2a4fc6b8047f05c9ac43e0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/49767752109c41b09db64719412c4502, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/45f203c3ad97418981535bfd97316596, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b54c26eaf7ef447da38539bc67296271, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/83ad5d097b0447d4a55e7a696a2e44b8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/cef668f78fc04575848a7fe76a1c7fe7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a3ac063ee20a4feea145fb20012ce1dd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f6fe0202237f4bc18aa3fd36c7dbb563, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/971bb98440944220af6a16f3f6ca829a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/2557c5aba8e14b6bbf22e462103fb472, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c727e1deab9f4db0925377020c949387, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/6e14b32f110a47c6a352e7219e432aa7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a96ee4b34a1948a8b41cb04f0793b08f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b4d372ab5ebd48569db27664fe914e69, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/305775a453d4424f8edb940038641015, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d9440c4bec0a4ef8af4f5048928afcf5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1183f985188c48149ff2a7a1f933e266, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b97b8e7569494b469c018275728fd063, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/3531697cc60246ef8f99e040d11c62ec, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/4bc99f1eeaf449d7be0bbf2a54d9bc9c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ba0a3dfc0a454959a06b57a3c732ea50, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f06fabe294dc470690de4a46880e081b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c6d050ec01124e90a0c73187675679f4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/632225da7ef540279f89e11377d1d227, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c28fcda6165f4e7ca215ea36d5730d7c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fa9bd8a8cac540068194114ca672ca38, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ae118c4717054034975d2c0922ac526e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/9acf876d38cf40f8b537a74b027c067e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3992efe1bc04723bcf9c61c1d44bd25, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/efda1a71102943898b87555feb0dfbd7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1765ed9ffc3245129ead02a8b9a3347e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/8fb8fbc234344ff385a901236cb42a02, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ce07157778f841ac977974d756209eb9] to archive 2023-03-26 11:54:05,728 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:54:05,729 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/eef069460f334e258c4565a5c28dcfb0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/eef069460f334e258c4565a5c28dcfb0 2023-03-26 11:54:05,730 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3059401bf68407887c87d879db9e3bd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3059401bf68407887c87d879db9e3bd 2023-03-26 11:54:05,731 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/819b69e5e9e54bc7a3bdbd56ba555c27 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/819b69e5e9e54bc7a3bdbd56ba555c27 2023-03-26 11:54:05,732 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fc84b9a174124525962bc7c430fad71e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fc84b9a174124525962bc7c430fad71e 2023-03-26 11:54:05,733 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/dad4d1c050db437189e8fc7311347412 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/dad4d1c050db437189e8fc7311347412 2023-03-26 11:54:05,734 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/59d13b30ce2a4fc6b8047f05c9ac43e0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/59d13b30ce2a4fc6b8047f05c9ac43e0 2023-03-26 11:54:05,735 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/49767752109c41b09db64719412c4502 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/49767752109c41b09db64719412c4502 2023-03-26 11:54:05,736 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/45f203c3ad97418981535bfd97316596 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/45f203c3ad97418981535bfd97316596 2023-03-26 11:54:05,737 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b54c26eaf7ef447da38539bc67296271 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b54c26eaf7ef447da38539bc67296271 2023-03-26 11:54:05,738 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/83ad5d097b0447d4a55e7a696a2e44b8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/83ad5d097b0447d4a55e7a696a2e44b8 2023-03-26 11:54:05,739 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/cef668f78fc04575848a7fe76a1c7fe7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/cef668f78fc04575848a7fe76a1c7fe7 2023-03-26 11:54:05,740 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a3ac063ee20a4feea145fb20012ce1dd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a3ac063ee20a4feea145fb20012ce1dd 2023-03-26 11:54:05,741 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f6fe0202237f4bc18aa3fd36c7dbb563 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f6fe0202237f4bc18aa3fd36c7dbb563 2023-03-26 11:54:05,742 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/971bb98440944220af6a16f3f6ca829a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/971bb98440944220af6a16f3f6ca829a 2023-03-26 11:54:05,743 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/2557c5aba8e14b6bbf22e462103fb472 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/2557c5aba8e14b6bbf22e462103fb472 2023-03-26 11:54:05,744 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c727e1deab9f4db0925377020c949387 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c727e1deab9f4db0925377020c949387 2023-03-26 11:54:05,746 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/6e14b32f110a47c6a352e7219e432aa7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/6e14b32f110a47c6a352e7219e432aa7 2023-03-26 11:54:05,747 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a96ee4b34a1948a8b41cb04f0793b08f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/a96ee4b34a1948a8b41cb04f0793b08f 2023-03-26 11:54:05,748 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b4d372ab5ebd48569db27664fe914e69 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b4d372ab5ebd48569db27664fe914e69 2023-03-26 11:54:05,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/305775a453d4424f8edb940038641015 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/305775a453d4424f8edb940038641015 2023-03-26 11:54:05,749 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d9440c4bec0a4ef8af4f5048928afcf5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d9440c4bec0a4ef8af4f5048928afcf5 2023-03-26 11:54:05,750 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1183f985188c48149ff2a7a1f933e266 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1183f985188c48149ff2a7a1f933e266 2023-03-26 11:54:05,751 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b97b8e7569494b469c018275728fd063 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/b97b8e7569494b469c018275728fd063 2023-03-26 11:54:05,752 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/3531697cc60246ef8f99e040d11c62ec to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/3531697cc60246ef8f99e040d11c62ec 2023-03-26 11:54:05,753 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/4bc99f1eeaf449d7be0bbf2a54d9bc9c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/4bc99f1eeaf449d7be0bbf2a54d9bc9c 2023-03-26 11:54:05,754 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ba0a3dfc0a454959a06b57a3c732ea50 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ba0a3dfc0a454959a06b57a3c732ea50 2023-03-26 11:54:05,755 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f06fabe294dc470690de4a46880e081b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f06fabe294dc470690de4a46880e081b 2023-03-26 11:54:05,756 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c6d050ec01124e90a0c73187675679f4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c6d050ec01124e90a0c73187675679f4 2023-03-26 11:54:05,757 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/632225da7ef540279f89e11377d1d227 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/632225da7ef540279f89e11377d1d227 2023-03-26 11:54:05,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c28fcda6165f4e7ca215ea36d5730d7c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/c28fcda6165f4e7ca215ea36d5730d7c 2023-03-26 11:54:05,759 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fa9bd8a8cac540068194114ca672ca38 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/fa9bd8a8cac540068194114ca672ca38 2023-03-26 11:54:05,760 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ae118c4717054034975d2c0922ac526e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ae118c4717054034975d2c0922ac526e 2023-03-26 11:54:05,762 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/9acf876d38cf40f8b537a74b027c067e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/9acf876d38cf40f8b537a74b027c067e 2023-03-26 11:54:05,763 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3992efe1bc04723bcf9c61c1d44bd25 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/f3992efe1bc04723bcf9c61c1d44bd25 2023-03-26 11:54:05,764 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/efda1a71102943898b87555feb0dfbd7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/efda1a71102943898b87555feb0dfbd7 2023-03-26 11:54:05,765 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1765ed9ffc3245129ead02a8b9a3347e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/1765ed9ffc3245129ead02a8b9a3347e 2023-03-26 11:54:05,766 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/8fb8fbc234344ff385a901236cb42a02 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/8fb8fbc234344ff385a901236cb42a02 2023-03-26 11:54:05,767 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ce07157778f841ac977974d756209eb9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ce07157778f841ac977974d756209eb9 2023-03-26 11:54:05,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2272920521104b91940bd67efcf03235, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/e05cd84161ef44e387562fc72e0203cd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/7880f87a96104dd3a0a58356400067e3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/b7e0c15f8f474080a444ae3ef65ae482, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/78fa7fc924ac443daa9506db962a4a45, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/bdbcc0f187d74a16a399901f0023cb6d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/5c040779725e4349b413a310cbef50ff, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/88fc80278ae74115aa6eb9b84cdcf6b0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3805cb941aa3458da5b2b542565c90db, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/af702c6ebc44475db23c9e3075ae7048, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/26bf5221f981491ab898c48b068f8fa0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/43519555bd10480aa63adf3afb09dbd4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2a5878a31d8740fa8b274cb1e877b99c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/54104fc6815b4dbeae4ab83a146665c3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9ab9159d974f45a5b59a4532b55f9706, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3e8049e2bf4441f5af6587b8303ad334, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/243fe59387f64b6ab2567fe314596f8b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/1c3c9fc7b7f5402287accf307880bf80, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/528cc522d1cd41edad95512ed89d0b89, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9426cfafe2a24ec7b8c19d121805259d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/240dea58dca342e68fcd4fe1931a66c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/586bdfa32db9424897d148ad90cb761b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/4ab198ae0760447f947e71dd0470f346, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/a405458e159b458d98244fb511210a71, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8ab1f0d0e05e47b9ac4e4c36e250910a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/c4c18af8af9d4e0b84b79188a10f4e3d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/cfdbf48c73284bbb8ccf78b748417119, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d87857a836704f2d96f5e1a80bb07633, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ec4e9d09f8d94cd8a8c382d29984a632, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d96843c82e3c4de39be6f51f65f70079, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/262320cef9ff4073801c693d4727f686, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/668c012162f34389a0510d42aae65702, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d5dd756327b64e80898f17d01d1197ba, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8fc3641c1cdb4552b911061a0d342453, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/0791125b17b043e1bd3bc5504116335d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ad73b30f423945bb88336d0c199241fd] to archive 2023-03-26 11:54:05,771 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:54:05,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2272920521104b91940bd67efcf03235 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2272920521104b91940bd67efcf03235 2023-03-26 11:54:05,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/e05cd84161ef44e387562fc72e0203cd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/e05cd84161ef44e387562fc72e0203cd 2023-03-26 11:54:05,774 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/7880f87a96104dd3a0a58356400067e3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/7880f87a96104dd3a0a58356400067e3 2023-03-26 11:54:05,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/b7e0c15f8f474080a444ae3ef65ae482 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/b7e0c15f8f474080a444ae3ef65ae482 2023-03-26 11:54:05,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/78fa7fc924ac443daa9506db962a4a45 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/78fa7fc924ac443daa9506db962a4a45 2023-03-26 11:54:05,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/bdbcc0f187d74a16a399901f0023cb6d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/bdbcc0f187d74a16a399901f0023cb6d 2023-03-26 11:54:05,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/5c040779725e4349b413a310cbef50ff to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/5c040779725e4349b413a310cbef50ff 2023-03-26 11:54:05,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/88fc80278ae74115aa6eb9b84cdcf6b0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/88fc80278ae74115aa6eb9b84cdcf6b0 2023-03-26 11:54:05,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3805cb941aa3458da5b2b542565c90db to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3805cb941aa3458da5b2b542565c90db 2023-03-26 11:54:05,781 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/af702c6ebc44475db23c9e3075ae7048 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/af702c6ebc44475db23c9e3075ae7048 2023-03-26 11:54:05,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/26bf5221f981491ab898c48b068f8fa0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/26bf5221f981491ab898c48b068f8fa0 2023-03-26 11:54:05,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/43519555bd10480aa63adf3afb09dbd4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/43519555bd10480aa63adf3afb09dbd4 2023-03-26 11:54:05,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2a5878a31d8740fa8b274cb1e877b99c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/2a5878a31d8740fa8b274cb1e877b99c 2023-03-26 11:54:05,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/54104fc6815b4dbeae4ab83a146665c3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/54104fc6815b4dbeae4ab83a146665c3 2023-03-26 11:54:05,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9ab9159d974f45a5b59a4532b55f9706 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9ab9159d974f45a5b59a4532b55f9706 2023-03-26 11:54:05,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3e8049e2bf4441f5af6587b8303ad334 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/3e8049e2bf4441f5af6587b8303ad334 2023-03-26 11:54:05,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/243fe59387f64b6ab2567fe314596f8b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/243fe59387f64b6ab2567fe314596f8b 2023-03-26 11:54:05,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/1c3c9fc7b7f5402287accf307880bf80 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/1c3c9fc7b7f5402287accf307880bf80 2023-03-26 11:54:05,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/528cc522d1cd41edad95512ed89d0b89 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/528cc522d1cd41edad95512ed89d0b89 2023-03-26 11:54:05,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9426cfafe2a24ec7b8c19d121805259d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/9426cfafe2a24ec7b8c19d121805259d 2023-03-26 11:54:05,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/240dea58dca342e68fcd4fe1931a66c1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/240dea58dca342e68fcd4fe1931a66c1 2023-03-26 11:54:05,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/586bdfa32db9424897d148ad90cb761b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/586bdfa32db9424897d148ad90cb761b 2023-03-26 11:54:05,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/4ab198ae0760447f947e71dd0470f346 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/4ab198ae0760447f947e71dd0470f346 2023-03-26 11:54:05,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/a405458e159b458d98244fb511210a71 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/a405458e159b458d98244fb511210a71 2023-03-26 11:54:05,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8ab1f0d0e05e47b9ac4e4c36e250910a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8ab1f0d0e05e47b9ac4e4c36e250910a 2023-03-26 11:54:05,797 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/c4c18af8af9d4e0b84b79188a10f4e3d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/c4c18af8af9d4e0b84b79188a10f4e3d 2023-03-26 11:54:05,798 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/cfdbf48c73284bbb8ccf78b748417119 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/cfdbf48c73284bbb8ccf78b748417119 2023-03-26 11:54:05,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d87857a836704f2d96f5e1a80bb07633 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d87857a836704f2d96f5e1a80bb07633 2023-03-26 11:54:05,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ec4e9d09f8d94cd8a8c382d29984a632 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ec4e9d09f8d94cd8a8c382d29984a632 2023-03-26 11:54:05,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d96843c82e3c4de39be6f51f65f70079 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d96843c82e3c4de39be6f51f65f70079 2023-03-26 11:54:05,801 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/262320cef9ff4073801c693d4727f686 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/262320cef9ff4073801c693d4727f686 2023-03-26 11:54:05,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/668c012162f34389a0510d42aae65702 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/668c012162f34389a0510d42aae65702 2023-03-26 11:54:05,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d5dd756327b64e80898f17d01d1197ba to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/d5dd756327b64e80898f17d01d1197ba 2023-03-26 11:54:05,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8fc3641c1cdb4552b911061a0d342453 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8fc3641c1cdb4552b911061a0d342453 2023-03-26 11:54:05,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/0791125b17b043e1bd3bc5504116335d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/0791125b17b043e1bd3bc5504116335d 2023-03-26 11:54:05,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ad73b30f423945bb88336d0c199241fd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/ad73b30f423945bb88336d0c199241fd 2023-03-26 11:54:05,810 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/recovered.edits/769.seqid, newMaxSeqId=769, maxSeqId=1 2023-03-26 11:54:05,811 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff. 2023-03-26 11:54:05,811 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 51042bc7573c44eb24dbd242f395f3ff: 2023-03-26 11:54:05,813 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:05,813 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=51042bc7573c44eb24dbd242f395f3ff, regionState=CLOSED 2023-03-26 11:54:05,814 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831645813"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831645813"}]},"ts":"1679831645813"} 2023-03-26 11:54:05,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-03-26 11:54:05,816 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 51042bc7573c44eb24dbd242f395f3ff, server=jenkins-hbase4.apache.org,38971,1679831615531 in 2.5790 sec 2023-03-26 11:54:05,817 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-03-26 11:54:05,818 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=51042bc7573c44eb24dbd242f395f3ff, UNASSIGN in 2.5850 sec 2023-03-26 11:54:05,818 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831645818"}]},"ts":"1679831645818"} 2023-03-26 11:54:05,819 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 11:54:05,820 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 11:54:05,822 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.6070 sec 2023-03-26 11:54:07,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-03-26 11:54:07,332 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-03-26 11:54:07,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 11:54:07,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,344 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,346 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-26 11:54:07,351 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:07,354 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/recovered.edits] 2023-03-26 11:54:07,357 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/4111df2a6db74c82b09abdb91f14ba60 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/4111df2a6db74c82b09abdb91f14ba60 2023-03-26 11:54:07,358 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5058a6268c8f4163bbac403f14338036 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/5058a6268c8f4163bbac403f14338036 2023-03-26 11:54:07,359 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9c3e091327c644dfae9262d32a6245fd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/A/9c3e091327c644dfae9262d32a6245fd 2023-03-26 11:54:07,362 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/66c6f1610c1b441d892579a2b811c9bd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/66c6f1610c1b441d892579a2b811c9bd 2023-03-26 11:54:07,363 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ac9444eaab174df98bd56bf96ca48f21 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/ac9444eaab174df98bd56bf96ca48f21 2023-03-26 11:54:07,365 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d3d589a17f334565b15079e89c606aa6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/B/d3d589a17f334565b15079e89c606aa6 2023-03-26 11:54:07,367 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/19655494548145d5a6ab179a638d4291 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/19655494548145d5a6ab179a638d4291 2023-03-26 11:54:07,369 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8316ad2923484b28a5829b034b18dfc4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/8316ad2923484b28a5829b034b18dfc4 2023-03-26 11:54:07,370 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/dc4cc8c7439443cf81b1be6aa7075d2e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/dc4cc8c7439443cf81b1be6aa7075d2e 2023-03-26 11:54:07,372 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/f84b596c03874fbf86230e6a49873a84 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/C/f84b596c03874fbf86230e6a49873a84 2023-03-26 11:54:07,374 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/recovered.edits/769.seqid to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff/recovered.edits/769.seqid 2023-03-26 11:54:07,375 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/51042bc7573c44eb24dbd242f395f3ff 2023-03-26 11:54:07,375 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:54:07,378 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,398 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 11:54:07,440 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 11:54:07,442 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,442 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 11:54:07,442 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679831647442"}]},"ts":"9223372036854775807"} 2023-03-26 11:54:07,444 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 11:54:07,444 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 51042bc7573c44eb24dbd242f395f3ff, NAME => 'TestAcidGuarantees,,1679831619174.51042bc7573c44eb24dbd242f395f3ff.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 11:54:07,444 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 11:54:07,444 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679831647444"}]},"ts":"9223372036854775807"} 2023-03-26 11:54:07,446 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 11:54:07,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-26 11:54:07,448 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,449 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 110 msec 2023-03-26 11:54:07,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-03-26 11:54:07,649 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-03-26 11:54:07,667 INFO [Listener at localhost/46409] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=234 (was 208) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd8a8e-shared-pool-18 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS:0;jenkins-hbase4:38971-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd8a8e-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_264488004_15 at /127.0.0.1:37600 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd8a8e-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x6bd8a8e-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_264488004_15 at /127.0.0.1:37638 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=560 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=403 (was 325) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=9046 (was 9634) 2023-03-26 11:54:07,678 INFO [Listener at localhost/46409] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234, OpenFileDescriptor=560, MaxFileDescriptor=60000, SystemLoadAverage=403, ProcessCount=169, AvailableMemoryMB=9046 2023-03-26 11:54:07,680 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:54:07,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:54:07,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:07,683 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:54:07,683 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-03-26 11:54:07,684 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:54:07,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 11:54:07,685 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:07,686 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 empty. 2023-03-26 11:54:07,686 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:07,686 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:54:07,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 11:54:07,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 11:54:08,095 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:54:08,097 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 2c5753ba51ccea59bc0c70ae6d9ee6c2, NAME => 'TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:54:08,102 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:54:08,102 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 2c5753ba51ccea59bc0c70ae6d9ee6c2, disabling compactions & flushes 2023-03-26 11:54:08,102 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,103 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,103 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. after waiting 0 ms 2023-03-26 11:54:08,103 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,103 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,103 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:08,105 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:54:08,106 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831648106"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831648106"}]},"ts":"1679831648106"} 2023-03-26 11:54:08,107 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:54:08,108 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:54:08,108 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831648108"}]},"ts":"1679831648108"} 2023-03-26 11:54:08,109 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 11:54:08,112 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, ASSIGN}] 2023-03-26 11:54:08,114 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, ASSIGN 2023-03-26 11:54:08,114 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:54:08,265 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:08,266 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831648265"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831648265"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831648265"}]},"ts":"1679831648265"} 2023-03-26 11:54:08,267 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:08,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 11:54:08,423 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2c5753ba51ccea59bc0c70ae6d9ee6c2, NAME => 'TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:54:08,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:54:08,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,424 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,426 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,427 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A 2023-03-26 11:54:08,427 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A 2023-03-26 11:54:08,427 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:08,427 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c5753ba51ccea59bc0c70ae6d9ee6c2 columnFamilyName A 2023-03-26 11:54:08,428 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(310): Store=2c5753ba51ccea59bc0c70ae6d9ee6c2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:08,428 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,429 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B 2023-03-26 11:54:08,429 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B 2023-03-26 11:54:08,429 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:08,430 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c5753ba51ccea59bc0c70ae6d9ee6c2 columnFamilyName B 2023-03-26 11:54:08,430 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(310): Store=2c5753ba51ccea59bc0c70ae6d9ee6c2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:08,430 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,432 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C 2023-03-26 11:54:08,432 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C 2023-03-26 11:54:08,432 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:08,432 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c5753ba51ccea59bc0c70ae6d9ee6c2 columnFamilyName C 2023-03-26 11:54:08,432 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(310): Store=2c5753ba51ccea59bc0c70ae6d9ee6c2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:08,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,434 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,435 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:54:08,437 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:08,438 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:54:08,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2c5753ba51ccea59bc0c70ae6d9ee6c2; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66810608, jitterRate=-0.0044443607330322266}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:54:08,439 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:08,439 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., pid=18, masterSystemTime=1679831648419 2023-03-26 11:54:08,441 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,441 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:08,441 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:08,442 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831648441"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831648441"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831648441"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831648441"}]},"ts":"1679831648441"} 2023-03-26 11:54:08,445 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-03-26 11:54:08,445 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 in 176 msec 2023-03-26 11:54:08,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-03-26 11:54:08,446 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, ASSIGN in 333 msec 2023-03-26 11:54:08,447 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:54:08,447 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831648447"}]},"ts":"1679831648447"} 2023-03-26 11:54:08,448 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 11:54:08,451 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:54:08,453 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 771 msec 2023-03-26 11:54:08,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-03-26 11:54:08,788 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-03-26 11:54:08,789 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x52274650 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:08,794 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7829b43e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:08,796 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:08,800 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57064, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:08,801 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:54:08,804 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:48650, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:54:08,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:54:08,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:54:08,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:09,230 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 11:54:09,231 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:54:09,231 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 11:54:09,235 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-26 11:54:09,248 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, REOPEN/MOVE}] 2023-03-26 11:54:09,249 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, REOPEN/MOVE 2023-03-26 11:54:09,250 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,250 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831649250"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831649250"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831649250"}]},"ts":"1679831649250"} 2023-03-26 11:54:09,252 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:09,404 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2c5753ba51ccea59bc0c70ae6d9ee6c2, disabling compactions & flushes 2023-03-26 11:54:09,405 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. after waiting 0 ms 2023-03-26 11:54:09,405 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-26 11:54:09,409 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,409 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:09,409 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 2c5753ba51ccea59bc0c70ae6d9ee6c2 to self. 2023-03-26 11:54:09,411 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,411 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=CLOSED 2023-03-26 11:54:09,412 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831649411"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831649411"}]},"ts":"1679831649411"} 2023-03-26 11:54:09,414 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-03-26 11:54:09,414 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 in 162 msec 2023-03-26 11:54:09,415 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=true 2023-03-26 11:54:09,565 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,566 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831649565"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831649565"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831649565"}]},"ts":"1679831649565"} 2023-03-26 11:54:09,567 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:09,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 2c5753ba51ccea59bc0c70ae6d9ee6c2, NAME => 'TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:54:09,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:54:09,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,722 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,724 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,725 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A 2023-03-26 11:54:09,725 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A 2023-03-26 11:54:09,726 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:09,728 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c5753ba51ccea59bc0c70ae6d9ee6c2 columnFamilyName A 2023-03-26 11:54:09,730 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(310): Store=2c5753ba51ccea59bc0c70ae6d9ee6c2/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:09,731 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,732 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B 2023-03-26 11:54:09,732 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B 2023-03-26 11:54:09,732 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:09,732 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c5753ba51ccea59bc0c70ae6d9ee6c2 columnFamilyName B 2023-03-26 11:54:09,733 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(310): Store=2c5753ba51ccea59bc0c70ae6d9ee6c2/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:09,733 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,734 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C 2023-03-26 11:54:09,734 DEBUG [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C 2023-03-26 11:54:09,734 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:09,734 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 2c5753ba51ccea59bc0c70ae6d9ee6c2 columnFamilyName C 2023-03-26 11:54:09,735 INFO [StoreOpener-2c5753ba51ccea59bc0c70ae6d9ee6c2-1] regionserver.HStore(310): Store=2c5753ba51ccea59bc0c70ae6d9ee6c2/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:09,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,737 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:54:09,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,740 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 2c5753ba51ccea59bc0c70ae6d9ee6c2; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60724829, jitterRate=-0.09512953460216522}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:54:09,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:09,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., pid=23, masterSystemTime=1679831649718 2023-03-26 11:54:09,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,742 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,743 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831649742"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831649742"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831649742"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831649742"}]},"ts":"1679831649742"} 2023-03-26 11:54:09,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-03-26 11:54:09,746 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 in 177 msec 2023-03-26 11:54:09,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-03-26 11:54:09,747 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, REOPEN/MOVE in 498 msec 2023-03-26 11:54:09,749 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-03-26 11:54:09,750 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 514 msec 2023-03-26 11:54:09,751 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 937 msec 2023-03-26 11:54:09,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-03-26 11:54:09,761 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x34135972 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,765 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33a47c05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,766 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,770 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,770 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,774 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,774 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,779 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,779 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,782 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,782 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,786 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,786 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,790 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,791 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,794 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,794 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:09,801 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:09,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:09,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:09,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:09,812 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:09,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:09,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:09,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:09,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:09,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:09,813 DEBUG [hconnection-0x752bab2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,813 DEBUG [hconnection-0x1afc6558-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,813 DEBUG [hconnection-0x23827ed7-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,815 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57074, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,815 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:09,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:09,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:09,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:09,816 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:09,816 DEBUG [hconnection-0xb5ab66b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:09,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:09,816 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:09,818 DEBUG [hconnection-0x634f7ff3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,818 DEBUG [hconnection-0x764abd48-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,822 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57090, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,822 DEBUG [hconnection-0x14124fc4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,822 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57088, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,823 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57098, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,823 DEBUG [hconnection-0x634b3ba0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,823 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:09,824 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57118, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,824 DEBUG [hconnection-0x74463726-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:09,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:09,825 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57112, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,826 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57136, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,828 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57114, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:09,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:09,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:09,828 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57134, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:09,835 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:09,835 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:09,837 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:09,837 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:09,849 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-26 11:54:09,863 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:09,864 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:09,864 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:09,864 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:09,864 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:09,865 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:09,867 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:09,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831649881, max timestamp=1679831649888Num uniques 859; 2023-03-26 11:54:09,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831649881, max timestamp=1679831649888Num uniques 859; 2023-03-26 11:54:09,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:54:09,892 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:54:09,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831649881, max timestamp=1679831649888Num uniques 859; 2023-03-26 11:54:09,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:09,918 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:09,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831709907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,921 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:09,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831709914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:09,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831709916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:09,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831709918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:09,922 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:09,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831709918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831710020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831710024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831710024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,031 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831710024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831710025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,231 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831710231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831710233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831710233, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,235 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831710234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,239 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831710237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,347 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303268b9db3d7e9384015afd4ce30b8994560 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268b9db3d7e9384015afd4ce30b8994560 2023-03-26 11:54:10,536 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831710534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831710539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831710539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831710541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:10,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831710543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:10,767 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=13, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/ce90e8047e95489d9ce73a997a6df16d 2023-03-26 11:54:11,041 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:11,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831711040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:11,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831711042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,044 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:11,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831711043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:11,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831711044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,047 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:11,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831711045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,181 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/795ff571ef934867a3ae7f74ae63eaae 2023-03-26 11:54:11,357 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 11:54:11,594 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/c99af99d356c4034b55912a1ff7524f3 2023-03-26 11:54:11,611 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/ce90e8047e95489d9ce73a997a6df16d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce90e8047e95489d9ce73a997a6df16d 2023-03-26 11:54:11,620 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce90e8047e95489d9ce73a997a6df16d, entries=150, sequenceid=13, filesize=25.1 K 2023-03-26 11:54:11,623 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/795ff571ef934867a3ae7f74ae63eaae as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/795ff571ef934867a3ae7f74ae63eaae 2023-03-26 11:54:11,629 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/795ff571ef934867a3ae7f74ae63eaae, entries=150, sequenceid=13, filesize=11.6 K 2023-03-26 11:54:11,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/c99af99d356c4034b55912a1ff7524f3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c99af99d356c4034b55912a1ff7524f3 2023-03-26 11:54:11,645 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c99af99d356c4034b55912a1ff7524f3, entries=150, sequenceid=13, filesize=11.6 K 2023-03-26 11:54:11,646 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 1807ms, sequenceid=13, compaction requested=false 2023-03-26 11:54:11,647 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 11:54:11,648 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:11,648 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:11,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:11,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:11,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:11,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:11,650 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:11,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:11,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:11,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:11,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:11,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3248ffba[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:11,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:11,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,654 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:11,654 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:11,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:11,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:11,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:11,659 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:11,659 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:11,659 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:11,659 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:11,659 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:11,662 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,662 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:11,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,662 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:11,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:11,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:11,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:11,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:11,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:11,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:11,669 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:11,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:11,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:11,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:11,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,669 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:11,670 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:11,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:11,670 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:11,770 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:11,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:11,775 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:11,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:11,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:11,784 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:11,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:11,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:11,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:11,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:11,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:11,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:11,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:11,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:11,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:11,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:11,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:11,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:11,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:11,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:11,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:11,788 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-26 11:54:11,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:11,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:11,789 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:11,789 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:11,789 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:11,789 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:11,834 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032682e005f163b941cba782067163ee1573 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682e005f163b941cba782067163ee1573 2023-03-26 11:54:11,853 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/d948bec8e0fb4917838b72062f79c986 2023-03-26 11:54:11,890 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/e8bf0ef727214d9582562531dbbad12a 2023-03-26 11:54:11,924 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/1b4debb80f2d4cb2b86c0c69d1ccaf67 2023-03-26 11:54:11,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/d948bec8e0fb4917838b72062f79c986 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/d948bec8e0fb4917838b72062f79c986 2023-03-26 11:54:11,938 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/d948bec8e0fb4917838b72062f79c986, entries=150, sequenceid=50, filesize=25.1 K 2023-03-26 11:54:11,939 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/e8bf0ef727214d9582562531dbbad12a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e8bf0ef727214d9582562531dbbad12a 2023-03-26 11:54:11,947 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e8bf0ef727214d9582562531dbbad12a, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 11:54:11,948 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/1b4debb80f2d4cb2b86c0c69d1ccaf67 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/1b4debb80f2d4cb2b86c0c69d1ccaf67 2023-03-26 11:54:11,962 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/1b4debb80f2d4cb2b86c0c69d1ccaf67, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 11:54:11,963 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=0 B/0 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 175ms, sequenceid=50, compaction requested=false 2023-03-26 11:54:11,964 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:11,964 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:11,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:11,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:11,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:11,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:11,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:11,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73469b31[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:11,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:11,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:11,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:11,971 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,971 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:11,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:11,973 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:11,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:11,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:11,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:11,976 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:11,976 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:11,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:11,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:11,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:11,976 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:11,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:11,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:11,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:11,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:11,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:11,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:11,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:11,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:11,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:11,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:11,983 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:11,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:11,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:11,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:11,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:11,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:11,983 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:11,984 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:11,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:11,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:11,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,084 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:12,087 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:12,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:12,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:12,094 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:12,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:12,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:12,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:12,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:12,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:12,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:12,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:12,094 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:12,094 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:12,095 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:12,097 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:12,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:12,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:12,104 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:12,104 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:12,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:12,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:12,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:12,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:12,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:12,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:12,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:12,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:12,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:12,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50cb8e03[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:12,123 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:12,123 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,124 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:12,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:12,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:12,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:12,125 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831652107, max timestamp=1679831652128Num uniques 859; 2023-03-26 11:54:12,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831652107, max timestamp=1679831652128Num uniques 859; 2023-03-26 11:54:12,130 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:12,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:12,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831652107, max timestamp=1679831652128Num uniques 859; 2023-03-26 11:54:12,131 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:12,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:12,133 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:12,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:12,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:12,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:12,133 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:12,138 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,138 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,138 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,141 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:12,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:12,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,145 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:12,145 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:12,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:12,146 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f831f336f4424ffaa3337d17582967a9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f831f336f4424ffaa3337d17582967a9 2023-03-26 11:54:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831712155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831712155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831712156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831712157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831712160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,167 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=60, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/a6a0aa3f0b634e5b9170525360609acf 2023-03-26 11:54:12,239 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/ae92a7b6b72c40b8a3b0395c6d3d39a6 2023-03-26 11:54:12,245 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:12,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:12,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831712262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831712262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831712262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831712262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,271 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831712267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:12,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:12,275 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:12,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:12,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:12,276 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,276 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,277 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:12,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,279 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:12,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:12,284 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:12,284 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:12,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:12,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:12,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:12,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:12,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:12,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:12,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:12,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:12,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:12,298 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,299 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:12,301 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cd57ff6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:12,301 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:12,301 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:12,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:12,302 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,302 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:12,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:12,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,304 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:12,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:12,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:12,306 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:12,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:12,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:12,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:12,307 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:12,309 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,309 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,309 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:12,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:12,328 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:12,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,329 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,329 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:12,329 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:12,429 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:12,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:12,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:12,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:12,441 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:12,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:12,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:12,442 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,442 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,443 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:12,444 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:12,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,445 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:12,445 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:12,445 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:12,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:12,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:12,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:12,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:12,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:12,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:12,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,455 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:12,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:12,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7995e4f9[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:12,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:12,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:12,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:12,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:12,458 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,458 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:12,461 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,461 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:12,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:12,464 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:12,464 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:12,464 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:12,464 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:12,464 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:12,465 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,465 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,465 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831712469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831712470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831712470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,472 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831712470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,473 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:12,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,474 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:12,474 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:12,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,479 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831712473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,574 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:12,576 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:12,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:12,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:12,578 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:12,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:12,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:12,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,580 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,580 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,581 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,581 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:12,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:12,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:12,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:12,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:12,582 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:12,582 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,582 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:12,583 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,583 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:12,583 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:12,583 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:12,583 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:12,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,586 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,586 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:12,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:12,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e4d77e9[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:12,587 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:12,588 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:12,589 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,589 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:12,589 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,590 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:12,590 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:12,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:12,594 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:12,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:12,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:12,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:12,594 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:12,595 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,595 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,596 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,596 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,601 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:12,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:12,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,602 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:12,602 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:12,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:12,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,699 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=60 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/6e23c91fc5dc460298e846227aee16f5 2023-03-26 11:54:12,702 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:12,704 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/a6a0aa3f0b634e5b9170525360609acf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a6a0aa3f0b634e5b9170525360609acf 2023-03-26 11:54:12,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:12,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:12,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:12,707 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:12,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:12,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:12,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,709 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a6a0aa3f0b634e5b9170525360609acf, entries=350, sequenceid=60, filesize=52.3 K 2023-03-26 11:54:12,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:12,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,711 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/ae92a7b6b72c40b8a3b0395c6d3d39a6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ae92a7b6b72c40b8a3b0395c6d3d39a6 2023-03-26 11:54:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:12,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,711 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:12,711 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:12,711 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:12,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:12,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:12,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:12,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:12,713 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,716 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:12,716 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:12,716 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:12,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:12,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:12,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:12,717 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ae92a7b6b72c40b8a3b0395c6d3d39a6, entries=150, sequenceid=60, filesize=11.6 K 2023-03-26 11:54:12,717 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,717 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,718 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/6e23c91fc5dc460298e846227aee16f5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6e23c91fc5dc460298e846227aee16f5 2023-03-26 11:54:12,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:12,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:12,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32fa4db5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:12,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:12,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,721 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:12,721 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,722 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:12,721 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:12,722 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:12,723 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6e23c91fc5dc460298e846227aee16f5, entries=150, sequenceid=60, filesize=11.6 K 2023-03-26 11:54:12,723 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 629ms, sequenceid=60, compaction requested=true 2023-03-26 11:54:12,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:12,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:12,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:12,724 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:12,724 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:12,724 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:12,725 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:12,725 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/A is initiating minor compaction (all files) 2023-03-26 11:54:12,725 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/A in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,725 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce90e8047e95489d9ce73a997a6df16d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/d948bec8e0fb4917838b72062f79c986, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a6a0aa3f0b634e5b9170525360609acf] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=102.5 K 2023-03-26 11:54:12,725 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:12,726 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/B is initiating minor compaction (all files) 2023-03-26 11:54:12,726 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/B in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,726 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/795ff571ef934867a3ae7f74ae63eaae, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e8bf0ef727214d9582562531dbbad12a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ae92a7b6b72c40b8a3b0395c6d3d39a6] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=34.8 K 2023-03-26 11:54:12,726 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ce90e8047e95489d9ce73a997a6df16d, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679831649849 2023-03-26 11:54:12,726 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 795ff571ef934867a3ae7f74ae63eaae, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679831649849 2023-03-26 11:54:12,726 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d948bec8e0fb4917838b72062f79c986, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831649899 2023-03-26 11:54:12,727 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e8bf0ef727214d9582562531dbbad12a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831649899 2023-03-26 11:54:12,727 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a6a0aa3f0b634e5b9170525360609acf, keycount=350, bloomtype=ROW, size=52.3 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679831652073 2023-03-26 11:54:12,727 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ae92a7b6b72c40b8a3b0395c6d3d39a6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679831652073 2023-03-26 11:54:12,728 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,730 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:12,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:12,732 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:12,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:12,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:12,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:12,732 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:12,736 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#B#compaction#131 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:12,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,738 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,741 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#A#compaction#132 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,746 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,746 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,747 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:12,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:12,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,747 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:12,747 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:12,760 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/ecd43671e2f94180a86aeaf9fbbf8fca as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ecd43671e2f94180a86aeaf9fbbf8fca 2023-03-26 11:54:12,767 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/B of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into ecd43671e2f94180a86aeaf9fbbf8fca(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:12,767 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:12,767 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/B, priority=13, startTime=1679831652724; duration=0sec 2023-03-26 11:54:12,767 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:12,767 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:12,771 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:12,771 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/C is initiating minor compaction (all files) 2023-03-26 11:54:12,771 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/C in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,771 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c99af99d356c4034b55912a1ff7524f3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/1b4debb80f2d4cb2b86c0c69d1ccaf67, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6e23c91fc5dc460298e846227aee16f5] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=34.8 K 2023-03-26 11:54:12,771 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c99af99d356c4034b55912a1ff7524f3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1679831649849 2023-03-26 11:54:12,772 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1b4debb80f2d4cb2b86c0c69d1ccaf67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831649899 2023-03-26 11:54:12,772 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6e23c91fc5dc460298e846227aee16f5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=60, earliestPutTs=1679831652073 2023-03-26 11:54:12,775 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-26 11:54:12,775 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:12,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:12,776 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:12,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:12,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:12,776 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:12,776 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:12,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831712781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,786 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831712782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831712784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,789 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831712785, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831712786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,801 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#C#compaction#133 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:12,821 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/43a269e6fcfd4f0491aef003c7dfd2a8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/43a269e6fcfd4f0491aef003c7dfd2a8 2023-03-26 11:54:12,827 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/C of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into 43a269e6fcfd4f0491aef003c7dfd2a8(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:12,827 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:12,827 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/C, priority=13, startTime=1679831652724; duration=0sec 2023-03-26 11:54:12,827 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:12,848 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:12,859 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:12,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:12,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:12,864 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:12,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:12,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:12,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,865 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:12,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,867 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:12,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,867 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:12,867 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:12,868 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:12,868 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,868 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:12,868 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:12,868 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:12,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:12,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:12,873 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1577bed3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:12,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:12,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:12,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:12,877 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:12,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:12,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:12,881 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:12,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:12,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:12,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:12,881 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:12,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:12,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:12,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,888 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:12,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:12,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831712888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831712891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831712892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:12,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:12,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:12,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:12,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:12,895 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:12,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:12,895 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:12,895 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:12,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:12,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:12,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:12,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831712892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:12,996 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:12,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,004 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,007 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,008 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,012 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,012 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,012 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,012 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,012 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,016 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,016 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,016 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,016 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:13,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:13,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:13,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,019 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,019 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,019 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:13,021 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@439322a8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:13,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:13,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,022 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:13,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:13,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:13,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:13,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:13,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:13,029 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:13,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:13,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:13,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:13,029 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:13,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:13,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,038 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:13,038 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:13,092 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831713092, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831713096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,097 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831713096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831713097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,138 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:13,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,147 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:13,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:13,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,153 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:13,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:13,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16b4e70c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:13,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:13,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:13,157 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:13,157 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:13,158 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:13,159 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,159 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:13,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:13,162 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:13,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:13,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:13,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:13,162 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:13,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:13,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:13,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,173 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,174 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,174 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:13,174 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:13,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,219 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032623a34e499b0a43c6ad6072284721f9fa to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032623a34e499b0a43c6ad6072284721f9fa 2023-03-26 11:54:13,274 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:13,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,278 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,280 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,281 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,282 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,282 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,282 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,282 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:13,282 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:13,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:13,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7643878a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:13,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:13,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:13,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:13,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:13,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:13,289 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:13,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831713289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,289 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,290 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,290 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:13,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:13,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:13,293 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:13,293 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:13,293 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:13,293 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:13,293 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:13,295 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,295 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,295 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,295 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,297 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:13,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:13,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,299 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:13,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,299 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:13,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,399 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831713394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,400 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:13,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,401 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831713399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,402 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831713399, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,402 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831713402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,403 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,406 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:13,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:13,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,409 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:13,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:13,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e6a3eb7[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:13,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:13,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:13,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:13,412 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,412 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:13,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,413 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,414 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:13,416 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:13,416 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:13,416 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:13,416 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:13,416 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:13,416 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:13,418 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,419 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,423 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:13,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:13,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,423 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,423 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:13,423 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:13,523 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:13,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,527 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:13,531 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:13,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,532 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:13,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:13,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@436d1b0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:13,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:13,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,536 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:13,536 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:13,536 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:13,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:13,538 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,538 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:13,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:13,540 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:13,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:13,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:13,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:13,540 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:13,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,543 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,544 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:13,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,550 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,551 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:13,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,551 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:13,551 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,610 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/1dae363c6c6f4548984524e9534bec94 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/1dae363c6c6f4548984524e9534bec94 2023-03-26 11:54:13,619 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/A of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into 1dae363c6c6f4548984524e9534bec94(size=25.2 K), total size for store is 25.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:13,619 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:13,619 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/A, priority=13, startTime=1679831652724; duration=0sec 2023-03-26 11:54:13,619 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:13,624 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/2179ab0afcab4d8bad105a109d407b87 2023-03-26 11:54:13,651 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:13,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,657 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,663 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,665 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/45bdcae1768e4ea4a82b4069762094c1 2023-03-26 11:54:13,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,666 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,667 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,667 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,669 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,669 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,669 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,669 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,669 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,670 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,670 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:13,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:13,671 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,671 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,672 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:13,673 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@358bcabd[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:13,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:13,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,674 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:13,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:13,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:13,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:13,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:13,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:13,682 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:13,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:13,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:13,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:13,682 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:13,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:13,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,684 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:13,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,686 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:13,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,689 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:13,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:13,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:13,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:13,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,690 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:13,690 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:13,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:13,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:13,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:13,692 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/ff9755454a9949979f6cde939b3746af 2023-03-26 11:54:13,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/2179ab0afcab4d8bad105a109d407b87 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/2179ab0afcab4d8bad105a109d407b87 2023-03-26 11:54:13,702 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/2179ab0afcab4d8bad105a109d407b87, entries=200, sequenceid=97, filesize=31.9 K 2023-03-26 11:54:13,703 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/45bdcae1768e4ea4a82b4069762094c1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/45bdcae1768e4ea4a82b4069762094c1 2023-03-26 11:54:13,707 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/45bdcae1768e4ea4a82b4069762094c1, entries=150, sequenceid=97, filesize=11.6 K 2023-03-26 11:54:13,708 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/ff9755454a9949979f6cde939b3746af as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/ff9755454a9949979f6cde939b3746af 2023-03-26 11:54:13,711 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/ff9755454a9949979f6cde939b3746af, entries=150, sequenceid=97, filesize=11.6 K 2023-03-26 11:54:13,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=40.25 KB/41220 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 937ms, sequenceid=97, compaction requested=false 2023-03-26 11:54:13,712 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:13,790 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:13,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:13,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:13,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:13,800 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:13,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:13,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:13,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:13,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:13,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:13,802 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:13,803 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:13,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:13,803 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:13,803 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:13,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:13,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:13,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:13,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:13,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:13,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:13,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:13,808 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:13,808 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:13,808 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 11:54:13,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:13,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:13,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:13,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:13,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:13,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:13,827 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032607e887f9f3f54c3687f2af86e0eb700a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032607e887f9f3f54c3687f2af86e0eb700a 2023-03-26 11:54:13,848 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5937e5996e3d4afb9b2f8b8dd6280ebe 2023-03-26 11:54:13,880 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/519895e7d71147cda118dbbff315d9bf 2023-03-26 11:54:13,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:13,903 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:13,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679831653914, max timestamp=1679831653920Num uniques 859; 2023-03-26 11:54:13,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679831653914, max timestamp=1679831653920Num uniques 859; 2023-03-26 11:54:13,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:13,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1679831653914, max timestamp=1679831653920Num uniques 859; 2023-03-26 11:54:13,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:13,922 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:13,944 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831713943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831713944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831713944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:13,947 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:13,947 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831713944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831714045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,050 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831714047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831714048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,051 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831714048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831714252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831714251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,253 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831714252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831714257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,293 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,294 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/6d778a2fbbbb41958415fb10f131e1b0 2023-03-26 11:54:14,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831714293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,299 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5937e5996e3d4afb9b2f8b8dd6280ebe as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5937e5996e3d4afb9b2f8b8dd6280ebe 2023-03-26 11:54:14,304 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5937e5996e3d4afb9b2f8b8dd6280ebe, entries=150, sequenceid=108, filesize=25.1 K 2023-03-26 11:54:14,305 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/519895e7d71147cda118dbbff315d9bf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/519895e7d71147cda118dbbff315d9bf 2023-03-26 11:54:14,309 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/519895e7d71147cda118dbbff315d9bf, entries=150, sequenceid=108, filesize=11.6 K 2023-03-26 11:54:14,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/6d778a2fbbbb41958415fb10f131e1b0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d778a2fbbbb41958415fb10f131e1b0 2023-03-26 11:54:14,318 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d778a2fbbbb41958415fb10f131e1b0, entries=150, sequenceid=108, filesize=11.6 K 2023-03-26 11:54:14,319 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=147.60 KB/151140 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 511ms, sequenceid=108, compaction requested=true 2023-03-26 11:54:14,319 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:14,319 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:14,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:14,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:14,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:14,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:14,324 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:14,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:14,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:14,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1e7451[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:14,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:14,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:14,331 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:14,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:14,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,332 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:14,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:14,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:14,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:14,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:14,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:14,335 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:14,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:14,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:14,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:14,335 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:14,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:14,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:14,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:14,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,339 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:14,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,342 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,342 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:14,342 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:14,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:14,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:14,342 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:14,342 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:14,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:14,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:14,342 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:14,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:14,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:14,342 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,342 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:14,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,343 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:14,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:14,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,443 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:14,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:14,445 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:14,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:14,446 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:14,446 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:14,446 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:14,446 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:14,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:14,448 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:14,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:14,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:14,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:14,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:14,449 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:14,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:14,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:14,449 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:14,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:14,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:14,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:14,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:14,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:14,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:14,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:14,450 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=147.60 KB heapSize=415.45 KB 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:14,450 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:14,474 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326bee46692a0aa4fe7bf206346b769d489 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326bee46692a0aa4fe7bf206346b769d489 2023-03-26 11:54:14,481 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=145, memsize=49.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5befde2e8966417aa3a8e5c1557ac24d 2023-03-26 11:54:14,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:14,555 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:14,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831714558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,561 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831714559, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,562 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831714561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,564 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831714562, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831714662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,664 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831714662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,665 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831714663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831714866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831714866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:14,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831714866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:14,898 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/54798523e6164c35a20903d2c3503919 2023-03-26 11:54:15,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831715069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831715170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,170 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831715170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,173 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831715171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,312 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=145 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/dd8bcebdd2b94cb18aedb09238ad63c7 2023-03-26 11:54:15,318 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5befde2e8966417aa3a8e5c1557ac24d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5befde2e8966417aa3a8e5c1557ac24d 2023-03-26 11:54:15,322 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5befde2e8966417aa3a8e5c1557ac24d, entries=150, sequenceid=145, filesize=25.3 K 2023-03-26 11:54:15,323 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/54798523e6164c35a20903d2c3503919 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/54798523e6164c35a20903d2c3503919 2023-03-26 11:54:15,326 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/54798523e6164c35a20903d2c3503919, entries=150, sequenceid=145, filesize=11.8 K 2023-03-26 11:54:15,327 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/dd8bcebdd2b94cb18aedb09238ad63c7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/dd8bcebdd2b94cb18aedb09238ad63c7 2023-03-26 11:54:15,331 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/dd8bcebdd2b94cb18aedb09238ad63c7, entries=150, sequenceid=145, filesize=11.8 K 2023-03-26 11:54:15,332 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~415.41 KB/425376, currentSize=33.54 KB/34350 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 882ms, sequenceid=145, compaction requested=true 2023-03-26 11:54:15,332 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:15,332 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:15,332 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:15,332 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:15,332 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,332 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:15,332 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:15,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,334 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:15,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:15,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:15,336 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@508e6683[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:15,336 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:15,336 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,337 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:15,337 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:15,337 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:15,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:15,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:15,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:15,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:15,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:15,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:15,342 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:15,342 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:15,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:15,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:15,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:15,342 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:15,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:15,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:15,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:15,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:15,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:15,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,345 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:15,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:15,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:15,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:15,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:15,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:15,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:15,351 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:15,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:15,351 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:15,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:15,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:15,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:15,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:15,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:15,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:15,463 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:15,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:15,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:15,466 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:15,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:15,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:15,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:15,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:15,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,468 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:15,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:15,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:15,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:15,469 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:15,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:15,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:15,470 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:15,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:15,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:15,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:15,470 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 11:54:15,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:15,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:15,488 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303260552bfee0c90416d92d04dd7c993dc67 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260552bfee0c90416d92d04dd7c993dc67 2023-03-26 11:54:15,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:15,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:15,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831655686, max timestamp=1679831655703Num uniques 859; 2023-03-26 11:54:15,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:54:15,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831655686, max timestamp=1679831655703Num uniques 859; 2023-03-26 11:54:15,705 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:54:15,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831655686, max timestamp=1679831655703Num uniques 859; 2023-03-26 11:54:15,707 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:54:15,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,746 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831715741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831715741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831715744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831715847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831715847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,851 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:15,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831715847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:15,898 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=153, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/082a7e0cab2e47cfaaad738ffdb67864 2023-03-26 11:54:16,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831716053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831716053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831716053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831716074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,310 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/e48c03fcdca34c7b8eb17b01f565c6d5 2023-03-26 11:54:16,313 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831716311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,315 DEBUG [Thread-950] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4153 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:54:16,334 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=153 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/c2d64e564688479a958841b90bf54279 2023-03-26 11:54:16,342 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/082a7e0cab2e47cfaaad738ffdb67864 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/082a7e0cab2e47cfaaad738ffdb67864 2023-03-26 11:54:16,351 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/082a7e0cab2e47cfaaad738ffdb67864, entries=150, sequenceid=153, filesize=25.3 K 2023-03-26 11:54:16,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/e48c03fcdca34c7b8eb17b01f565c6d5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e48c03fcdca34c7b8eb17b01f565c6d5 2023-03-26 11:54:16,357 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e48c03fcdca34c7b8eb17b01f565c6d5, entries=150, sequenceid=153, filesize=11.8 K 2023-03-26 11:54:16,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831716358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831716358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,360 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831716358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,360 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/c2d64e564688479a958841b90bf54279 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c2d64e564688479a958841b90bf54279 2023-03-26 11:54:16,365 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c2d64e564688479a958841b90bf54279, entries=150, sequenceid=153, filesize=11.8 K 2023-03-26 11:54:16,367 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 897ms, sequenceid=153, compaction requested=true 2023-03-26 11:54:16,367 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:16,367 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:16,367 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:16,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:16,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:16,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:16,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:16,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:16,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62078673[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:16,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:16,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,374 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:16,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:16,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:16,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:16,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:16,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:16,376 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:16,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:16,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:16,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:16,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:16,381 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:16,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:16,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:16,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:16,381 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:16,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:16,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:16,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:16,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:16,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:16,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:16,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:16,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:16,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:16,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:16,390 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:16,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:16,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:16,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:16,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:16,390 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:16,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:16,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:16,491 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:16,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:16,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:16,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:16,495 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:16,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:16,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:16,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:16,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:16,497 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:16,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:16,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:16,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:16,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:16,498 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:16,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:16,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:16,499 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:16,499 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:16,499 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-26 11:54:16,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:16,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:16,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:16,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:16,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:16,500 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:16,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:16,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:16,875 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831716869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,875 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831716871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831716871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,920 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326eaa087bf8dc54bd88aa97ace71ec7672 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326eaa087bf8dc54bd88aa97ace71ec7672 2023-03-26 11:54:16,940 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/ce210e64ff6746fda5db149df4518e22 2023-03-26 11:54:16,972 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/e055344ca8f241cc8a545c573e8f98c0 2023-03-26 11:54:16,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831716977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,980 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831716977, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:16,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831716979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:16,999 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/3f2e91edd77d4f1c93a8555f4a103b5f 2023-03-26 11:54:17,008 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/ce210e64ff6746fda5db149df4518e22 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce210e64ff6746fda5db149df4518e22 2023-03-26 11:54:17,014 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce210e64ff6746fda5db149df4518e22, entries=150, sequenceid=190, filesize=25.3 K 2023-03-26 11:54:17,019 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/e055344ca8f241cc8a545c573e8f98c0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e055344ca8f241cc8a545c573e8f98c0 2023-03-26 11:54:17,032 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e055344ca8f241cc8a545c573e8f98c0, entries=150, sequenceid=190, filesize=11.8 K 2023-03-26 11:54:17,033 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/3f2e91edd77d4f1c93a8555f4a103b5f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3f2e91edd77d4f1c93a8555f4a103b5f 2023-03-26 11:54:17,038 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3f2e91edd77d4f1c93a8555f4a103b5f, entries=150, sequenceid=190, filesize=11.8 K 2023-03-26 11:54:17,040 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 540ms, sequenceid=190, compaction requested=true 2023-03-26 11:54:17,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:17,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:17,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:17,042 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:17,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:17,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:17,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:17,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@93276ac[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:17,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:17,045 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,046 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:17,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:17,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:17,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:17,048 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:17,048 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:17,048 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:17,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:17,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:17,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:17,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:17,053 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:17,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:17,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:17,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:17,053 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:17,060 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,060 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:17,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:17,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,061 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:17,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:17,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:17,064 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:17,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:17,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:17,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:17,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:17,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:17,066 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:17,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:17,067 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:17,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:17,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,067 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:17,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:17,067 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,067 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:17,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:17,067 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:17,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:17,067 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:17,067 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,167 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:17,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:17,170 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:17,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:17,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:17,171 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:17,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:17,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:17,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:17,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,172 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:17,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:17,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:17,173 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:17,173 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:17,173 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:17,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:17,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:17,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:17,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:17,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:17,175 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:17,175 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:17,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:17,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:17,185 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:17,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831657200, max timestamp=1679831657207Num uniques 859; 2023-03-26 11:54:17,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831657200, max timestamp=1679831657207Num uniques 859; 2023-03-26 11:54:17,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:17,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:17,209 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831657200, max timestamp=1679831657207Num uniques 859; 2023-03-26 11:54:17,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:17,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1679831657223, max timestamp=1679831657241Num uniques 1118; 2023-03-26 11:54:17,243 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=19138, suffix heap size=144240,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=418 2023-03-26 11:54:17,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1679831657223, max timestamp=1679831657241Num uniques 1118; 2023-03-26 11:54:17,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=19138, suffix heap size=144240,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=418 2023-03-26 11:54:17,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=418, cellSize=19138, totalHeapSize=43712, min timestamp=1679831657223, max timestamp=1679831657241Num uniques 1118; 2023-03-26 11:54:17,244 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=51198, new segment data size=19138, suffix heap size=144240,new segment heap size=43712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1118, new segment cells count=418 2023-03-26 11:54:17,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831717293, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,301 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831717294, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831717295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831717402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831717402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831717402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,591 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326adb2d866645a45b9b23552b97999e1f3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326adb2d866645a45b9b23552b97999e1f3 2023-03-26 11:54:17,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831717606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,611 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831717607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,613 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831717607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831717912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831717914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:17,917 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:17,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831717914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,001 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=198, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5224a3126ffc4e0396c225603300ae1c 2023-03-26 11:54:18,094 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:18,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831718093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,096 DEBUG [Thread-952] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:54:18,414 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/b9b834844f5f490f852893f7a7885df2 2023-03-26 11:54:18,420 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831718418, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,421 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:18,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831718419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,423 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:18,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831718421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,825 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=198 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/f336c8428c414a888a520f28543ce029 2023-03-26 11:54:18,831 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5224a3126ffc4e0396c225603300ae1c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5224a3126ffc4e0396c225603300ae1c 2023-03-26 11:54:18,835 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5224a3126ffc4e0396c225603300ae1c, entries=150, sequenceid=198, filesize=25.3 K 2023-03-26 11:54:18,842 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/b9b834844f5f490f852893f7a7885df2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b9b834844f5f490f852893f7a7885df2 2023-03-26 11:54:18,847 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b9b834844f5f490f852893f7a7885df2, entries=150, sequenceid=198, filesize=11.8 K 2023-03-26 11:54:18,848 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/f336c8428c414a888a520f28543ce029 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/f336c8428c414a888a520f28543ce029 2023-03-26 11:54:18,853 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/f336c8428c414a888a520f28543ce029, entries=150, sequenceid=198, filesize=11.8 K 2023-03-26 11:54:18,854 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 1679ms, sequenceid=198, compaction requested=true 2023-03-26 11:54:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:18,854 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:18,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:18,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25222727[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:18,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:18,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:18,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:18,863 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,863 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:18,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:18,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:18,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:18,867 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:18,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:18,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:18,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:18,867 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:18,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:18,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:18,871 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:18,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:18,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:18,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:18,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:18,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:18,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:18,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:18,875 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:18,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,875 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:18,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:18,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:18,975 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:18,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:18,978 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:18,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:18,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:18,979 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:18,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:18,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:18,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:18,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:18,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:18,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:18,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:18,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:18,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:18,982 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:18,983 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:18,983 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:18,983 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:18,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:18,984 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:18,984 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=154.31 KB heapSize=420.47 KB 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:18,984 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:18,998 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326fcb88aa4d12e45e68002828cc7d2d031 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326fcb88aa4d12e45e68002828cc7d2d031 2023-03-26 11:54:19,403 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=250, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/aafedb47ba1640838926fdcdf9966ea4 2023-03-26 11:54:19,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:19,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:19,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831719433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831719440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,443 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831719441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831719541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,546 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831719541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831719544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57118 deadline: 1679831719747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,750 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57134 deadline: 1679831719748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,756 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:19,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57074 deadline: 1679831719754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,816 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/0c1f0a6941154d509d3df1004bd33521 2023-03-26 11:54:19,831 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=250 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/33275713e30a463a8a42384fcf7dd575 2023-03-26 11:54:19,835 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/aafedb47ba1640838926fdcdf9966ea4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/aafedb47ba1640838926fdcdf9966ea4 2023-03-26 11:54:19,839 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/aafedb47ba1640838926fdcdf9966ea4, entries=150, sequenceid=250, filesize=25.3 K 2023-03-26 11:54:19,840 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/0c1f0a6941154d509d3df1004bd33521 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/0c1f0a6941154d509d3df1004bd33521 2023-03-26 11:54:19,844 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/0c1f0a6941154d509d3df1004bd33521, entries=150, sequenceid=250, filesize=11.8 K 2023-03-26 11:54:19,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/33275713e30a463a8a42384fcf7dd575 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/33275713e30a463a8a42384fcf7dd575 2023-03-26 11:54:19,850 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/33275713e30a463a8a42384fcf7dd575, entries=150, sequenceid=250, filesize=11.8 K 2023-03-26 11:54:19,851 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~420.42 KB/430512, currentSize=40.25 KB/41220 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 867ms, sequenceid=250, compaction requested=true 2023-03-26 11:54:19,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:19,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:19,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:19,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:19,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:19,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:19,859 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bbe77e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:19,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:19,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:19,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:19,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:19,862 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,862 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:19,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:19,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:19,865 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:19,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:19,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:19,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:19,868 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:19,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:19,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:19,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:19,868 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:19,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:19,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:19,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:19,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:19,871 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:19,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:19,873 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:19,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:19,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,876 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:19,876 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:19,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:19,876 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:19,876 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:19,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:19,876 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:19,876 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:19,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:19,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:19,877 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:19,877 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:19,877 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:19,877 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,877 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:19,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:19,877 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:19,977 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:19,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:19,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:19,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:19,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:19,981 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:19,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:19,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:19,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:19,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:19,998 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:19,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:19,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:19,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:19,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:20,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:20,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:20,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:20,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:20,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,000 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 11:54:20,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:20,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:20,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:20,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:20,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:20,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:20,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:20,018 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263647cf1eea994114af7d4fcdfd0484bc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263647cf1eea994114af7d4fcdfd0484bc 2023-03-26 11:54:20,024 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=259, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/b2054a2e12ab448dac5d2cee4c1ebc7c 2023-03-26 11:54:20,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:20,056 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831660073, max timestamp=1679831660093Num uniques 859; 2023-03-26 11:54:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831660073, max timestamp=1679831660093Num uniques 859; 2023-03-26 11:54:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831660073, max timestamp=1679831660093Num uniques 859; 2023-03-26 11:54:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:20,095 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:20,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831720122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831720123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831720125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,236 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831720227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831720228, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,237 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831720231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57136 deadline: 1679831720353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,361 DEBUG [Thread-950] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8200 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:54:20,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831720438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,443 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/c0717a3091094a58b2668ade2bf60aac 2023-03-26 11:54:20,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831720439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831720439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,458 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=259 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/4cf7a32af0c943bb9f8a1f12b4659c4d 2023-03-26 11:54:20,466 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/b2054a2e12ab448dac5d2cee4c1ebc7c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/b2054a2e12ab448dac5d2cee4c1ebc7c 2023-03-26 11:54:20,471 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/b2054a2e12ab448dac5d2cee4c1ebc7c, entries=150, sequenceid=259, filesize=25.4 K 2023-03-26 11:54:20,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/c0717a3091094a58b2668ade2bf60aac as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c0717a3091094a58b2668ade2bf60aac 2023-03-26 11:54:20,477 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c0717a3091094a58b2668ade2bf60aac, entries=150, sequenceid=259, filesize=11.9 K 2023-03-26 11:54:20,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/4cf7a32af0c943bb9f8a1f12b4659c4d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/4cf7a32af0c943bb9f8a1f12b4659c4d 2023-03-26 11:54:20,482 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/4cf7a32af0c943bb9f8a1f12b4659c4d, entries=150, sequenceid=259, filesize=11.9 K 2023-03-26 11:54:20,483 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=140.89 KB/144270 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 483ms, sequenceid=259, compaction requested=true 2023-03-26 11:54:20,483 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:20,483 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:20,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:20,485 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:20,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76635b15[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:20,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:20,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,488 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:20,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:20,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:20,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:20,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:20,492 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:20,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:20,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:20,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:20,492 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:20,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:20,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:20,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:20,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:20,499 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:20,499 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:20,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:20,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:20,599 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:20,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:20,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:20,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:20,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:20,603 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:20,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:20,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:20,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,606 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:20,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:20,607 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,607 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,607 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,608 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:20,608 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:20,608 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:20,609 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=140.89 KB heapSize=395.53 KB 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:20,609 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:20,635 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326def4bb54df6742e2a2fa5df7847a89f0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326def4bb54df6742e2a2fa5df7847a89f0 2023-03-26 11:54:20,646 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=295, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/65986d747815493a9180d5771ba3cd98 2023-03-26 11:54:20,662 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/dea1aab82e364b79979048fe1b7e12b6 2023-03-26 11:54:20,684 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=295 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/c91b6f08daa44be69d4ce3f4c2f1908a 2023-03-26 11:54:20,693 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/65986d747815493a9180d5771ba3cd98 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/65986d747815493a9180d5771ba3cd98 2023-03-26 11:54:20,698 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/65986d747815493a9180d5771ba3cd98, entries=150, sequenceid=295, filesize=25.4 K 2023-03-26 11:54:20,700 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/dea1aab82e364b79979048fe1b7e12b6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/dea1aab82e364b79979048fe1b7e12b6 2023-03-26 11:54:20,704 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/dea1aab82e364b79979048fe1b7e12b6, entries=150, sequenceid=295, filesize=11.9 K 2023-03-26 11:54:20,705 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/c91b6f08daa44be69d4ce3f4c2f1908a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c91b6f08daa44be69d4ce3f4c2f1908a 2023-03-26 11:54:20,716 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c91b6f08daa44be69d4ce3f4c2f1908a, entries=150, sequenceid=295, filesize=11.9 K 2023-03-26 11:54:20,717 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~395.48 KB/404976, currentSize=0 B/0 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 108ms, sequenceid=295, compaction requested=true 2023-03-26 11:54:20,717 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:20,717 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:20,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,720 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,720 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:20,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:20,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f3ac2b0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:20,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:20,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,725 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:20,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:20,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:20,727 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:20,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,727 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,728 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:20,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:20,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:20,732 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:20,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:20,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:20,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:20,732 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:20,736 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,736 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,738 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:20,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:20,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:20,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,742 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:20,742 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:20,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:20,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:20,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,748 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:20,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:20,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:20,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:20,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:20,831 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:20,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:20,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:20,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:20,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:20,832 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:20,832 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:20,842 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:20,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:20,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:20,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:20,854 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:20,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:20,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:20,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:20,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:20,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:20,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,858 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:20,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,859 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:20,859 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:20,859 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:20,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:20,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:20,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:20,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:20,859 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,859 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:20,859 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:20,859 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:20,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:20,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:20,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:20,861 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326d3774e38990247daa981f94b05c1f46f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d3774e38990247daa981f94b05c1f46f 2023-03-26 11:54:20,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,862 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:20,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7953857e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:20,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:20,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:20,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:20,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:20,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:20,876 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:20,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:20,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:20,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:20,877 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:20,878 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,878 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:20,878 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,879 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:20,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,880 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:20,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:20,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:20,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:20,883 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:20,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,883 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:20,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:20,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:20,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831720890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831660887, max timestamp=1679831660892Num uniques 859; 2023-03-26 11:54:20,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831660887, max timestamp=1679831660892Num uniques 859; 2023-03-26 11:54:20,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:20,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831660887, max timestamp=1679831660892Num uniques 859; 2023-03-26 11:54:20,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:20,894 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:20,942 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831720934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831720940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,984 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:20,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:20,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:20,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:20,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:20,987 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:20,987 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,987 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:20,987 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:20,987 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,987 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:20,989 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:20,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:20,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:20,991 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:20,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:20,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:20,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:20,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:20,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:20,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:20,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:20,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:20,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:20,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:20,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,994 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:20,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:20,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:20,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59840a47[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:20,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:20,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,997 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:20,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:20,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:20,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:20,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:20,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:20,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:20,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831720994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:21,002 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:21,002 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:21,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:21,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:21,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:21,002 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:21,004 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,004 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,004 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:21,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:21,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,011 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:21,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,011 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:21,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831721043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831721043, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,111 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:21,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:21,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:21,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:21,114 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:21,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:21,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:21,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,117 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,117 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:21,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,117 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:21,118 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:21,118 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:21,118 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,118 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:21,118 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:21,118 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:21,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:21,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:21,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:21,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35534991[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:21,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:21,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,124 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:21,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:21,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,125 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:21,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:21,128 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:21,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:21,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:21,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:21,128 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:21,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,133 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,133 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,133 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:21,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:21,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,134 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:21,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,133 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,134 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:21,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831721201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,234 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:21,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:21,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:21,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:21,238 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:21,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:21,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:21,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,241 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,241 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,241 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,241 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:21,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,242 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:21,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:21,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:21,242 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:21,243 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:21,243 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:21,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:21,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:21,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c0522c9[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:21,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:21,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,247 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:21,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:21,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,249 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,252 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831721250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:21,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:21,252 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:21,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:21,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:21,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:21,252 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:21,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831721251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,257 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:21,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:21,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,262 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:21,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:21,262 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:21,280 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=306, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/9a20a02221f349c7832324d5ede537af 2023-03-26 11:54:21,363 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:21,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:21,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:21,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:21,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:21,367 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:21,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:21,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:21,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,369 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,370 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,370 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:21,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:21,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:21,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:21,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:21,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:21,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:21,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:21,379 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,380 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:21,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29982642[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:21,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:21,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,383 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:21,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:21,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:21,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:21,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:21,387 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:21,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:21,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:21,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:21,387 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:21,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:21,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:21,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:21,394 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:21,394 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:21,494 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:21,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:21,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:21,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:21,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:21,497 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:21,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:21,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:21,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:21,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,500 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:21,500 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:21,500 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:21,500 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:21,501 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,501 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:21,501 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:21,501 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:21,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:21,504 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,504 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:21,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:21,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5037d704[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:21,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,508 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831721504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:21,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,510 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,510 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:21,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:21,513 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:21,513 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:21,513 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:21,513 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:21,513 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:21,514 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,514 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,514 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,517 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,520 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:21,520 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:21,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:21,520 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,521 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:21,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,520 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,521 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:21,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831721554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:21,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831721557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,621 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:21,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:21,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:21,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:21,624 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:21,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:21,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:21,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,626 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,626 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,627 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,627 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:21,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,627 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:21,628 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:21,628 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:21,628 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,628 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:21,628 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:21,628 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:21,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:21,629 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,629 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:21,629 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:21,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,630 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:21,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f871352[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:21,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:21,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,633 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:21,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:21,633 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,634 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:21,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:21,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:21,637 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:21,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:21,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:21,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:21,637 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:21,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:21,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:21,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,639 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:21,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,641 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:21,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:21,645 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:21,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:21,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:21,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,646 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:21,646 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:21,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:21,651 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:21,651 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:21,691 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/ed625797b4ad4114a27ea6c6b0848948 2023-03-26 11:54:21,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=306 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/cbe265f7df104f89a5dbecf5a095050f 2023-03-26 11:54:21,716 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/9a20a02221f349c7832324d5ede537af as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a20a02221f349c7832324d5ede537af 2023-03-26 11:54:21,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a20a02221f349c7832324d5ede537af, entries=350, sequenceid=306, filesize=52.9 K 2023-03-26 11:54:21,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/ed625797b4ad4114a27ea6c6b0848948 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ed625797b4ad4114a27ea6c6b0848948 2023-03-26 11:54:21,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ed625797b4ad4114a27ea6c6b0848948, entries=150, sequenceid=306, filesize=11.9 K 2023-03-26 11:54:21,730 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/cbe265f7df104f89a5dbecf5a095050f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cbe265f7df104f89a5dbecf5a095050f 2023-03-26 11:54:21,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cbe265f7df104f89a5dbecf5a095050f, entries=150, sequenceid=306, filesize=11.9 K 2023-03-26 11:54:21,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 907ms, sequenceid=306, compaction requested=true 2023-03-26 11:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:21,738 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-03-26 11:54:21,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:21,738 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-03-26 11:54:21,739 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:21,750 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 265803 starting at candidate #0 after considering 44 permutations with 44 in ratio 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/A is initiating minor compaction 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120263 starting at candidate #0 after considering 44 permutations with 44 in ratio 2023-03-26 11:54:21,751 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/A in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/B is initiating minor compaction 2023-03-26 11:54:21,751 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/B in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,751 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/1dae363c6c6f4548984524e9534bec94, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/2179ab0afcab4d8bad105a109d407b87, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5937e5996e3d4afb9b2f8b8dd6280ebe, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5befde2e8966417aa3a8e5c1557ac24d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/082a7e0cab2e47cfaaad738ffdb67864, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce210e64ff6746fda5db149df4518e22, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5224a3126ffc4e0396c225603300ae1c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/aafedb47ba1640838926fdcdf9966ea4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/b2054a2e12ab448dac5d2cee4c1ebc7c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/65986d747815493a9180d5771ba3cd98] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=259.6 K 2023-03-26 11:54:21,751 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ecd43671e2f94180a86aeaf9fbbf8fca, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/45bdcae1768e4ea4a82b4069762094c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/519895e7d71147cda118dbbff315d9bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/54798523e6164c35a20903d2c3503919, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e48c03fcdca34c7b8eb17b01f565c6d5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e055344ca8f241cc8a545c573e8f98c0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b9b834844f5f490f852893f7a7885df2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/0c1f0a6941154d509d3df1004bd33521, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c0717a3091094a58b2668ade2bf60aac, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/dea1aab82e364b79979048fe1b7e12b6] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=117.4 K 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1dae363c6c6f4548984524e9534bec94, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=60 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ecd43671e2f94180a86aeaf9fbbf8fca, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2179ab0afcab4d8bad105a109d407b87, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=97 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 45bdcae1768e4ea4a82b4069762094c1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5937e5996e3d4afb9b2f8b8dd6280ebe, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=108 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 519895e7d71147cda118dbbff315d9bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5befde2e8966417aa3a8e5c1557ac24d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=145 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 54798523e6164c35a20903d2c3503919, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e48c03fcdca34c7b8eb17b01f565c6d5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e055344ca8f241cc8a545c573e8f98c0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b9b834844f5f490f852893f7a7885df2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 0c1f0a6941154d509d3df1004bd33521, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c0717a3091094a58b2668ade2bf60aac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dea1aab82e364b79979048fe1b7e12b6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295 2023-03-26 11:54:21,751 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:21,751 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 082a7e0cab2e47cfaaad738ffdb67864, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=153 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ce210e64ff6746fda5db149df4518e22, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5224a3126ffc4e0396c225603300ae1c, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=198 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting aafedb47ba1640838926fdcdf9966ea4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=250 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b2054a2e12ab448dac5d2cee4c1ebc7c, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=259 2023-03-26 11:54:21,752 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 65986d747815493a9180d5771ba3cd98, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=295 2023-03-26 11:54:21,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:21,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:21,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:21,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:21,758 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:21,758 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,758 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:21,758 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:21,759 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,759 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,765 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:21,765 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:21,765 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:21,765 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,766 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:21,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:21,766 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:21,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:21,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:21,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:21,769 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:21,769 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:21,769 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:21,769 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-26 11:54:21,769 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:21,769 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:21,769 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:21,771 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:21,785 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#B#compaction#164 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:21,803 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#A#compaction#166 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:21,809 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032621256f809c444a698623bb4db14cb344 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032621256f809c444a698623bb4db14cb344 2023-03-26 11:54:22,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:22,012 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:22,053 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831722050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831722061, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,072 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831722063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57090 deadline: 1679831722105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,110 DEBUG [Thread-952] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:54:22,156 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831722154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,205 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/c28e17fc90bd4d088dd4cd4c466c3b17 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c28e17fc90bd4d088dd4cd4c466c3b17 2023-03-26 11:54:22,210 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/B of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into c28e17fc90bd4d088dd4cd4c466c3b17(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:22,210 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:22,210 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/B, priority=5, startTime=1679831661738; duration=0sec 2023-03-26 11:54:22,211 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:22,211 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 11 store files, 0 compacting, 11 eligible, 16 blocking 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 120263 starting at candidate #0 after considering 44 permutations with 44 in ratio 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/C is initiating minor compaction 2023-03-26 11:54:22,218 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/C in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:22,218 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/43a269e6fcfd4f0491aef003c7dfd2a8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/ff9755454a9949979f6cde939b3746af, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d778a2fbbbb41958415fb10f131e1b0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/dd8bcebdd2b94cb18aedb09238ad63c7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c2d64e564688479a958841b90bf54279, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3f2e91edd77d4f1c93a8555f4a103b5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/f336c8428c414a888a520f28543ce029, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/33275713e30a463a8a42384fcf7dd575, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/4cf7a32af0c943bb9f8a1f12b4659c4d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c91b6f08daa44be69d4ce3f4c2f1908a] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=117.4 K 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 43a269e6fcfd4f0491aef003c7dfd2a8, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=60 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ff9755454a9949979f6cde939b3746af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6d778a2fbbbb41958415fb10f131e1b0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dd8bcebdd2b94cb18aedb09238ad63c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=145 2023-03-26 11:54:22,218 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c2d64e564688479a958841b90bf54279, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=153 2023-03-26 11:54:22,219 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3f2e91edd77d4f1c93a8555f4a103b5f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190 2023-03-26 11:54:22,219 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f336c8428c414a888a520f28543ce029, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=198 2023-03-26 11:54:22,219 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 33275713e30a463a8a42384fcf7dd575, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=250 2023-03-26 11:54:22,219 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 4cf7a32af0c943bb9f8a1f12b4659c4d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=259 2023-03-26 11:54:22,219 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c91b6f08daa44be69d4ce3f4c2f1908a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=295 2023-03-26 11:54:22,226 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=341, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/74d758bd32fc40f6a94cb2199984fe52 2023-03-26 11:54:22,245 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#C#compaction#168 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:22,253 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/74ff946c56534f73bcaff9c2bc11672c 2023-03-26 11:54:22,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831722357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,645 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/f2472432617c410c9c54c478d7a8f46c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f2472432617c410c9c54c478d7a8f46c 2023-03-26 11:54:22,650 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/A of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into f2472432617c410c9c54c478d7a8f46c(size=25.9 K), total size for store is 78.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:22,650 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:22,650 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/A, priority=5, startTime=1679831661738; duration=0sec 2023-03-26 11:54:22,650 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:22,659 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/91352d576704495a8de3fd4a1e14fbe2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/91352d576704495a8de3fd4a1e14fbe2 2023-03-26 11:54:22,663 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/C of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into 91352d576704495a8de3fd4a1e14fbe2(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:22,663 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:22,663 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/C, priority=5, startTime=1679831661739; duration=0sec 2023-03-26 11:54:22,664 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:22,664 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=341 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/249bbafcdc7448f6b28eadfd9353d07f 2023-03-26 11:54:22,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:22,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831722662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,670 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/74d758bd32fc40f6a94cb2199984fe52 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/74d758bd32fc40f6a94cb2199984fe52 2023-03-26 11:54:22,674 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/74d758bd32fc40f6a94cb2199984fe52, entries=150, sequenceid=341, filesize=25.4 K 2023-03-26 11:54:22,675 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/74ff946c56534f73bcaff9c2bc11672c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/74ff946c56534f73bcaff9c2bc11672c 2023-03-26 11:54:22,679 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/74ff946c56534f73bcaff9c2bc11672c, entries=150, sequenceid=341, filesize=11.9 K 2023-03-26 11:54:22,680 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/249bbafcdc7448f6b28eadfd9353d07f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/249bbafcdc7448f6b28eadfd9353d07f 2023-03-26 11:54:22,684 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/249bbafcdc7448f6b28eadfd9353d07f, entries=150, sequenceid=341, filesize=11.9 K 2023-03-26 11:54:22,685 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 916ms, sequenceid=341, compaction requested=true 2023-03-26 11:54:22,685 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:22,685 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:22,686 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:22,686 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,686 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:22,686 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:22,688 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,688 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:22,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:22,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:22,691 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1af16501[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:22,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:22,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,693 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:22,693 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:22,693 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,693 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:22,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,694 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:22,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:22,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:22,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:22,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:22,697 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:22,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:22,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:22,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:22,697 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:22,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:22,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:22,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:22,699 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:22,701 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:22,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:22,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:22,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:22,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:22,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:22,703 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,703 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:22,703 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:22,703 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:22,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:22,803 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:22,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:22,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:22,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:22,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:22,806 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:22,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:22,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:22,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:22,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:22,809 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,809 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:22,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:22,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:22,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:22,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:22,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:22,811 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:22,811 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:22,811 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:22,811 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:22,811 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:22,812 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:22,812 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:22,827 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303269a4dbcb97705402da0d624331a825245 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269a4dbcb97705402da0d624331a825245 2023-03-26 11:54:22,836 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=355, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/6b2de7b5abcc444a8cdcd9a2e7d6f458 2023-03-26 11:54:23,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:23,077 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:23,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831723106, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831663094, max timestamp=1679831663108Num uniques 859; 2023-03-26 11:54:23,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:23,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831663094, max timestamp=1679831663108Num uniques 859; 2023-03-26 11:54:23,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:23,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831663094, max timestamp=1679831663108Num uniques 859; 2023-03-26 11:54:23,112 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:23,146 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831723143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,173 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831723171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 181 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831723213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,246 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/b3fde9723d9f4817b9e2126669a01562 2023-03-26 11:54:23,248 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831723247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,263 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=355 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/b2fe06b1f1b942ad8815f05385e8de8c 2023-03-26 11:54:23,275 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/6b2de7b5abcc444a8cdcd9a2e7d6f458 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6b2de7b5abcc444a8cdcd9a2e7d6f458 2023-03-26 11:54:23,281 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6b2de7b5abcc444a8cdcd9a2e7d6f458, entries=150, sequenceid=355, filesize=25.4 K 2023-03-26 11:54:23,282 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/b3fde9723d9f4817b9e2126669a01562 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b3fde9723d9f4817b9e2126669a01562 2023-03-26 11:54:23,286 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b3fde9723d9f4817b9e2126669a01562, entries=150, sequenceid=355, filesize=11.9 K 2023-03-26 11:54:23,287 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/b2fe06b1f1b942ad8815f05385e8de8c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/b2fe06b1f1b942ad8815f05385e8de8c 2023-03-26 11:54:23,291 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/b2fe06b1f1b942ad8815f05385e8de8c, entries=150, sequenceid=355, filesize=11.9 K 2023-03-26 11:54:23,292 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 480ms, sequenceid=355, compaction requested=true 2023-03-26 11:54:23,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:23,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:23,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:23,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:23,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:23,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,297 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,298 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,298 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:23,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7233fa2b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:23,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:23,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:23,302 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:23,302 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,302 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:23,303 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,303 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:23,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:23,306 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:23,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:23,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:23,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:23,306 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:23,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:23,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:23,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:23,312 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,313 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,313 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:23,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,413 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:23,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:23,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:23,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:23,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:23,418 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:23,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:23,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:23,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:23,419 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-26 11:54:23,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:23,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:23,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:23,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:23,419 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:23,419 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:23,420 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:23,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,422 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:23,422 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:23,422 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:23,422 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:23,423 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,423 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:23,423 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:23,423 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:23,423 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:23,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,424 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,425 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:23,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1cd28c9e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:23,428 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:23,428 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,430 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:23,430 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:23,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:23,433 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:23,433 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,434 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:23,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,437 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263e669cc91f8d400e9b46a41abdd17601 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263e669cc91f8d400e9b46a41abdd17601 2023-03-26 11:54:23,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:23,438 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:23,438 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:23,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:23,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:23,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:23,439 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:23,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,442 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,447 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/36b1881a477b49218447fc23e2691565 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,451 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:23,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:23,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:23,452 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:23,452 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:23,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831723451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831723451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/5a81ec54a6724812bf1f379cd2925093 2023-03-26 11:54:23,498 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/55f98d7f5aa84b4490b23dfdb37d260c 2023-03-26 11:54:23,502 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/36b1881a477b49218447fc23e2691565 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/36b1881a477b49218447fc23e2691565 2023-03-26 11:54:23,507 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/36b1881a477b49218447fc23e2691565, entries=200, sequenceid=390, filesize=32.3 K 2023-03-26 11:54:23,508 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/5a81ec54a6724812bf1f379cd2925093 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/5a81ec54a6724812bf1f379cd2925093 2023-03-26 11:54:23,513 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/5a81ec54a6724812bf1f379cd2925093, entries=150, sequenceid=390, filesize=11.9 K 2023-03-26 11:54:23,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/55f98d7f5aa84b4490b23dfdb37d260c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/55f98d7f5aa84b4490b23dfdb37d260c 2023-03-26 11:54:23,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/55f98d7f5aa84b4490b23dfdb37d260c, entries=150, sequenceid=390, filesize=11.9 K 2023-03-26 11:54:23,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 101ms, sequenceid=390, compaction requested=true 2023-03-26 11:54:23,520 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:23,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:23,520 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:54:23,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:23,520 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 11:54:23,520 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:54:23,522 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 165882 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:54:23,522 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/A is initiating minor compaction (all files) 2023-03-26 11:54:23,522 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/A in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,522 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f2472432617c410c9c54c478d7a8f46c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a20a02221f349c7832324d5ede537af, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/74d758bd32fc40f6a94cb2199984fe52, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6b2de7b5abcc444a8cdcd9a2e7d6f458, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/36b1881a477b49218447fc23e2691565] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=162.0 K 2023-03-26 11:54:23,523 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f2472432617c410c9c54c478d7a8f46c, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679831660120 2023-03-26 11:54:23,523 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:54:23,523 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/B is initiating minor compaction (all files) 2023-03-26 11:54:23,523 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/B in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,523 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c28e17fc90bd4d088dd4cd4c466c3b17, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ed625797b4ad4114a27ea6c6b0848948, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/74ff946c56534f73bcaff9c2bc11672c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b3fde9723d9f4817b9e2126669a01562, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/5a81ec54a6724812bf1f379cd2925093] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=59.9 K 2023-03-26 11:54:23,524 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9a20a02221f349c7832324d5ede537af, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679831660812 2023-03-26 11:54:23,525 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c28e17fc90bd4d088dd4cd4c466c3b17, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679831660120 2023-03-26 11:54:23,525 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 74d758bd32fc40f6a94cb2199984fe52, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679831660909 2023-03-26 11:54:23,525 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ed625797b4ad4114a27ea6c6b0848948, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679831660812 2023-03-26 11:54:23,526 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 74ff946c56534f73bcaff9c2bc11672c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679831660909 2023-03-26 11:54:23,526 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 6b2de7b5abcc444a8cdcd9a2e7d6f458, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679831662036 2023-03-26 11:54:23,526 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b3fde9723d9f4817b9e2126669a01562, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679831662036 2023-03-26 11:54:23,527 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 36b1881a477b49218447fc23e2691565, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679831663137 2023-03-26 11:54:23,527 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5a81ec54a6724812bf1f379cd2925093, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679831663137 2023-03-26 11:54:23,547 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#B#compaction#176 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:23,552 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:23,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:23,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:23,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:23,559 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:54:23,559 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:23,559 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:23,559 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:23,559 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:23,559 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:23,559 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:23,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:23,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:23,564 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:23,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:23,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:23,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,566 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:23,570 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#A#compaction#177 average throughput is 0.76 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:23,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:23,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:23,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:23,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:23,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:23,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:23,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:23,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:23,584 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:23,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:23,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:23,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8811d5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:23,590 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:23,590 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,591 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,591 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:23,592 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:23,592 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,592 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:23,593 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,594 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:23,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:23,598 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:23,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:23,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:23,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:23,598 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:23,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,607 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303264f18bdf4069f4f9094ea1a4693339afe to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264f18bdf4069f4f9094ea1a4693339afe 2023-03-26 11:54:23,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,608 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:23,608 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,608 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:23,608 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:23,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:23,614 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/a2c70d6bac834e648e0259266590ad7a 2023-03-26 11:54:23,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/60a532cb64264a2db86ce0a097bcd954 2023-03-26 11:54:23,644 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831663607, max timestamp=1679831663643Num uniques 859; 2023-03-26 11:54:23,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:23,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831663607, max timestamp=1679831663643Num uniques 859; 2023-03-26 11:54:23,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831663607, max timestamp=1679831663643Num uniques 859; 2023-03-26 11:54:23,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:23,645 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:23,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831723695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,708 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:23,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:23,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:23,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:23,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:23,711 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:23,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:23,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:23,712 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,712 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:23,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:23,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:23,715 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,715 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:23,715 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:23,715 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:23,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,717 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,717 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:23,719 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4ac0aec8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:23,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:23,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,720 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:23,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:23,720 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:23,723 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:23,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:23,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:23,725 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:23,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:23,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:23,725 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:23,725 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:23,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,728 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,729 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,731 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:23,731 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,731 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:23,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,732 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,732 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,732 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,758 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831723757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,803 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:23,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831723797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,832 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:23,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:23,834 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:23,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:23,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:23,836 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:23,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:23,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:23,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,836 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,838 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,838 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:23,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,839 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:23,839 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:23,839 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:23,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:23,839 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,840 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:23,840 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:23,840 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:23,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:23,843 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,843 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:23,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:23,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27cff719[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:23,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:23,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,847 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:23,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:23,847 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,847 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:23,848 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:23,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:23,850 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:23,850 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:23,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:23,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:23,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:23,851 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:23,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,858 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:23,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:23,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,861 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,861 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:23,862 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:23,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,962 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:23,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:23,964 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:23,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:23,965 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:23,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:23,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:23,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,967 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:23,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:23,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,968 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:23,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:23,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:23,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:23,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:23,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:23,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:23,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:23,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:23,977 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:23,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff13902[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:23,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:23,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,981 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:23,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:23,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:23,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:23,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:23,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,984 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/cf657b9a4fbc4de090c3de5962c503ef as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/cf657b9a4fbc4de090c3de5962c503ef 2023-03-26 11:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:23,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:23,985 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:23,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:23,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:23,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:23,986 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:23,987 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,987 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:23,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:23,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,987 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,988 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/B of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into cf657b9a4fbc4de090c3de5962c503ef(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:23,989 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:23,989 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/B, priority=11, startTime=1679831663520; duration=0sec 2023-03-26 11:54:23,989 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:23,989 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:54:23,991 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61374 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:54:23,991 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/C is initiating minor compaction (all files) 2023-03-26 11:54:23,991 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/C in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:23,992 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/91352d576704495a8de3fd4a1e14fbe2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cbe265f7df104f89a5dbecf5a095050f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/249bbafcdc7448f6b28eadfd9353d07f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/b2fe06b1f1b942ad8815f05385e8de8c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/55f98d7f5aa84b4490b23dfdb37d260c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=59.9 K 2023-03-26 11:54:23,992 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 91352d576704495a8de3fd4a1e14fbe2, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=295, earliestPutTs=1679831660120 2023-03-26 11:54:23,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:23,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:23,993 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting cbe265f7df104f89a5dbecf5a095050f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=306, earliestPutTs=1679831660812 2023-03-26 11:54:23,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:23,994 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 249bbafcdc7448f6b28eadfd9353d07f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=341, earliestPutTs=1679831660909 2023-03-26 11:54:23,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,995 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b2fe06b1f1b942ad8815f05385e8de8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=355, earliestPutTs=1679831662036 2023-03-26 11:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:23,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,995 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 55f98d7f5aa84b4490b23dfdb37d260c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679831663137 2023-03-26 11:54:23,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:23,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:23,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:23,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:23,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:24,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:23,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:24,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:24,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:24,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:24,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,000 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:24,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:24,000 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:24,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:24,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:24,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:24,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:24,008 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#C#compaction#181 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:24,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831724007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,008 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/6180179ac282476f83868539f6f97b43 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6180179ac282476f83868539f6f97b43 2023-03-26 11:54:24,041 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/A of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into 6180179ac282476f83868539f6f97b43(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:24,041 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:24,041 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/A, priority=11, startTime=1679831663520; duration=0sec 2023-03-26 11:54:24,042 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:24,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/aeb9119e1b1d4dd79d80eeba74a24979 2023-03-26 11:54:24,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/a2c70d6bac834e648e0259266590ad7a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a2c70d6bac834e648e0259266590ad7a 2023-03-26 11:54:24,048 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/be942c29768a43b796fb365d3682c331 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/be942c29768a43b796fb365d3682c331 2023-03-26 11:54:24,052 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a2c70d6bac834e648e0259266590ad7a, entries=200, sequenceid=401, filesize=32.3 K 2023-03-26 11:54:24,053 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/60a532cb64264a2db86ce0a097bcd954 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/60a532cb64264a2db86ce0a097bcd954 2023-03-26 11:54:24,054 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/C of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into be942c29768a43b796fb365d3682c331(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:24,054 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:24,054 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/C, priority=11, startTime=1679831663520; duration=0sec 2023-03-26 11:54:24,054 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:24,063 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/60a532cb64264a2db86ce0a097bcd954, entries=150, sequenceid=401, filesize=11.9 K 2023-03-26 11:54:24,064 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/aeb9119e1b1d4dd79d80eeba74a24979 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/aeb9119e1b1d4dd79d80eeba74a24979 2023-03-26 11:54:24,071 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/aeb9119e1b1d4dd79d80eeba74a24979, entries=150, sequenceid=401, filesize=11.9 K 2023-03-26 11:54:24,071 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 512ms, sequenceid=401, compaction requested=false 2023-03-26 11:54:24,072 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:24,100 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:24,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:24,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:24,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:24,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:24,105 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:24,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:24,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:24,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:24,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:24,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:24,107 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:24,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:24,107 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,107 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:24,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:24,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:24,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:24,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:24,109 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:24,109 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:24,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:24,109 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:24,109 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 11:54:24,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:24,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:24,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:24,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:24,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:24,110 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:24,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:24,179 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:24,201 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831724200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,266 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831724263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,306 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831724303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831724310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831724507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,542 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032678761bedd9464e59a30a67fe76f4cb95 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032678761bedd9464e59a30a67fe76f4cb95 2023-03-26 11:54:24,548 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=440, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/f64487f7b8d44c2fb20847ab227339ac 2023-03-26 11:54:24,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831724811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,819 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:24,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831724819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:24,965 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/57633d8eb37243f7abff55e4aa0a3e37 2023-03-26 11:54:24,987 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=440 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/d228456721504044a1fe658ab78d1731 2023-03-26 11:54:24,997 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/f64487f7b8d44c2fb20847ab227339ac as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f64487f7b8d44c2fb20847ab227339ac 2023-03-26 11:54:25,003 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f64487f7b8d44c2fb20847ab227339ac, entries=150, sequenceid=440, filesize=25.4 K 2023-03-26 11:54:25,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/57633d8eb37243f7abff55e4aa0a3e37 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/57633d8eb37243f7abff55e4aa0a3e37 2023-03-26 11:54:25,008 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/57633d8eb37243f7abff55e4aa0a3e37, entries=150, sequenceid=440, filesize=11.9 K 2023-03-26 11:54:25,009 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/d228456721504044a1fe658ab78d1731 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/d228456721504044a1fe658ab78d1731 2023-03-26 11:54:25,012 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/d228456721504044a1fe658ab78d1731, entries=150, sequenceid=440, filesize=11.9 K 2023-03-26 11:54:25,013 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 904ms, sequenceid=440, compaction requested=true 2023-03-26 11:54:25,013 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:25,013 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:25,013 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:25,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:25,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:25,018 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,018 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:25,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:25,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:25,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34ec9390[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:25,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:25,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,021 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:25,021 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:25,021 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:25,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:25,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:25,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:25,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:25,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:25,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:25,027 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:25,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:25,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:25,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:25,027 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:25,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:25,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:25,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:25,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:25,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:25,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:25,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,034 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:25,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:25,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:25,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:25,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:25,035 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:25,035 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:25,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:25,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:25,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:25,135 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:25,135 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:25,138 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:25,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:25,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:25,140 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:25,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:25,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:25,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:25,141 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,141 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:25,143 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:25,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:25,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:25,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:25,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:25,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:25,152 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:25,155 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:25,155 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:25,155 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:25,156 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:54:25,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:25,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:25,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:25,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:25,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:25,156 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:25,172 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032682722cb793a24460bf0b9069fc12e2c0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682722cb793a24460bf0b9069fc12e2c0 2023-03-26 11:54:25,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:25,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:25,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,330 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831665324, max timestamp=1679831665328Num uniques 859; 2023-03-26 11:54:25,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831725330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,331 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:25,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831665324, max timestamp=1679831665328Num uniques 859; 2023-03-26 11:54:25,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:25,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831665324, max timestamp=1679831665328Num uniques 859; 2023-03-26 11:54:25,332 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:25,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831725392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,432 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831725431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,501 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831725498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,578 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=451, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/90668bbadc6f48fda564933de008c260 2023-03-26 11:54:25,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831725634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831725702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,831 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831725827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,940 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:25,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831725938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:25,992 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b 2023-03-26 11:54:26,004 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/bae97e3faf5040a5b2dd767b920f88ee 2023-03-26 11:54:26,009 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/90668bbadc6f48fda564933de008c260 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/90668bbadc6f48fda564933de008c260 2023-03-26 11:54:26,012 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831726008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,013 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/90668bbadc6f48fda564933de008c260, entries=150, sequenceid=451, filesize=25.4 K 2023-03-26 11:54:26,014 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b 2023-03-26 11:54:26,017 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b, entries=150, sequenceid=451, filesize=11.9 K 2023-03-26 11:54:26,018 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/bae97e3faf5040a5b2dd767b920f88ee as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/bae97e3faf5040a5b2dd767b920f88ee 2023-03-26 11:54:26,023 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/bae97e3faf5040a5b2dd767b920f88ee, entries=150, sequenceid=451, filesize=11.9 K 2023-03-26 11:54:26,023 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 868ms, sequenceid=451, compaction requested=true 2023-03-26 11:54:26,024 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:26,024 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:26,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:26,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,026 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,026 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:26,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b95dd2b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:26,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:26,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:26,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:26,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:26,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:26,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:26,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:26,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:26,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:26,032 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:26,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:26,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:26,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:26,032 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:26,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,035 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,037 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:26,040 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:26,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:26,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,040 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:26,040 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:26,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,140 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:26,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:26,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:26,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:26,144 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,147 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,148 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:26,148 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:26,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:26,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:26,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:26,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:26,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:26,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:26,151 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 11:54:26,152 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:26,152 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:26,152 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:26,152 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:26,152 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:26,152 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:26,212 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303262d4c48a79a3b4c3185c2444735975cf9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262d4c48a79a3b4c3185c2444735975cf9 2023-03-26 11:54:26,252 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=487, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/de5b100d32db4319b39b3f29b3881ccf 2023-03-26 11:54:26,298 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/2d45e6a6e13549f29dc2b914cfa405f0 2023-03-26 11:54:26,322 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=487 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/3641ce19d46b424ebb69c71338a8759f 2023-03-26 11:54:26,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/de5b100d32db4319b39b3f29b3881ccf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/de5b100d32db4319b39b3f29b3881ccf 2023-03-26 11:54:26,332 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/de5b100d32db4319b39b3f29b3881ccf, entries=150, sequenceid=487, filesize=25.4 K 2023-03-26 11:54:26,333 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/2d45e6a6e13549f29dc2b914cfa405f0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/2d45e6a6e13549f29dc2b914cfa405f0 2023-03-26 11:54:26,344 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/2d45e6a6e13549f29dc2b914cfa405f0, entries=150, sequenceid=487, filesize=11.9 K 2023-03-26 11:54:26,346 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/3641ce19d46b424ebb69c71338a8759f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3641ce19d46b424ebb69c71338a8759f 2023-03-26 11:54:26,356 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3641ce19d46b424ebb69c71338a8759f, entries=150, sequenceid=487, filesize=11.9 K 2023-03-26 11:54:26,357 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=0 B/0 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 206ms, sequenceid=487, compaction requested=true 2023-03-26 11:54:26,358 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:26,358 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:26,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:26,365 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:26,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ba9a7f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:26,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:26,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,373 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:26,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:26,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:26,375 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:26,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:26,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:26,388 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:26,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:26,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:26,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:26,388 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:26,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,391 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,393 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:26,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:26,401 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,402 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:26,402 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:26,502 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:26,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:26,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:26,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:26,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:26,506 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,510 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:26,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:26,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:26,510 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:26,510 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,510 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:26,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:26,511 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:26,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:26,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:26,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:26,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:26,511 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:26,511 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:26,511 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:26,512 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:26,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:26,514 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,514 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,514 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,514 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:26,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:26,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a58e23[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:26,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,517 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:26,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:26,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:26,518 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,518 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:26,519 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:26,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:26,523 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:26,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:26,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:26,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:26,523 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:26,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,527 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,531 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,531 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:26,531 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:26,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:26,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,532 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:26,532 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:26,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831666543, max timestamp=1679831666568Num uniques 859; 2023-03-26 11:54:26,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831666543, max timestamp=1679831666568Num uniques 859; 2023-03-26 11:54:26,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831666543, max timestamp=1679831666568Num uniques 859; 2023-03-26 11:54:26,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:26,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:26,570 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:26,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831726598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831726599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,632 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:26,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:26,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:26,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:26,635 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:26,635 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,635 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:26,635 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:26,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,639 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:26,639 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:26,639 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:26,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:26,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:26,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:26,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:26,639 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,639 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:26,639 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:26,640 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:26,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:26,641 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,641 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,641 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:26,641 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:26,644 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5505ed14[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:26,644 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:26,644 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,646 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:26,646 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:26,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:26,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:26,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:26,653 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:26,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:26,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:26,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:26,653 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:26,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,657 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,660 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:26,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:26,660 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:26,660 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:26,660 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:26,660 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,660 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831726702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,705 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831726703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,760 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:26,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:26,762 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:26,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:26,764 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:26,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,768 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:26,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:26,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:26,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:26,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:26,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:26,769 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:26,770 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,770 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,771 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:26,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac0e1a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:26,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:26,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,773 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:26,773 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:26,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:26,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:26,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,779 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:26,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:26,781 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:26,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:26,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:26,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:26,781 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:26,782 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,782 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,782 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:26,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:26,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:26,786 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:26,786 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:26,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,886 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:26,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:26,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:26,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:26,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:26,890 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:26,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:26,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:26,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,891 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,892 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,892 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:26,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,893 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:26,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,893 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:26,894 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:26,894 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:26,894 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,894 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:26,894 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:26,894 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:26,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:26,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:26,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77320512[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:26,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:26,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,900 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,900 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:26,900 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:26,900 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:26,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:26,901 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,901 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:26,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:26,904 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:26,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:26,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:26,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:26,904 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:26,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:26,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,906 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:26,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:26,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:26,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:26,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:26,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,915 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:26,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:26,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:26,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:26,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:26,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:26,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:26,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831726907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,916 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:26,916 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:26,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:26,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831726908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:26,927 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f104dd87de8644a0a9d4765ed934c49f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f104dd87de8644a0a9d4765ed934c49f 2023-03-26 11:54:26,942 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=497, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5286bdedb8214a3da0cc2c77a0724d55 2023-03-26 11:54:27,016 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,019 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,021 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ecf5941[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,027 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,027 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,027 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,032 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,032 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,033 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,034 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,035 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,039 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,039 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,039 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,139 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,143 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,146 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,146 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,146 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,146 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,146 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@245862c1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,152 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,152 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,153 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,156 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,156 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,157 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,157 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,157 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,158 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,158 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,166 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,166 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:27,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831727219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,223 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:27,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831727219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,266 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,269 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,270 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,270 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,271 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,272 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,272 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,272 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,273 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,273 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,273 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,273 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,274 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,276 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44e4648e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,276 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,278 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,279 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,280 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,282 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,282 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,283 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,288 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,288 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,353 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/9ba6a6cb203f43519d6a7a4e6efe6ec2 2023-03-26 11:54:27,388 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,391 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,400 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,400 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,401 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,401 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,402 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,404 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2399e395[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,407 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,409 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,410 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,410 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,413 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,416 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:27,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,416 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,417 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,417 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,517 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,518 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,520 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,521 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@738b3f6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,531 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,532 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,532 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,533 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,541 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:27,542 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,542 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,642 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,645 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,647 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,648 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,648 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,649 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,650 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,650 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,650 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,651 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@609187ba[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,655 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,655 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,659 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,659 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,659 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,659 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,659 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,659 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,661 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,664 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,665 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,665 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,727 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:27,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831727723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,728 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:27,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831727728, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,765 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,766 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/7917cd30445048ec8bec409638dd2b4a 2023-03-26 11:54:27,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,769 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,769 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,769 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,769 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,770 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,770 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,772 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,773 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/5286bdedb8214a3da0cc2c77a0724d55 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5286bdedb8214a3da0cc2c77a0724d55 2023-03-26 11:54:27,773 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:27,773 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,773 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,774 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,774 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,774 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,774 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,776 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5286bdedb8214a3da0cc2c77a0724d55, entries=350, sequenceid=497, filesize=52.9 K 2023-03-26 11:54:27,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,779 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/9ba6a6cb203f43519d6a7a4e6efe6ec2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/9ba6a6cb203f43519d6a7a4e6efe6ec2 2023-03-26 11:54:27,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/9ba6a6cb203f43519d6a7a4e6efe6ec2, entries=150, sequenceid=497, filesize=11.9 K 2023-03-26 11:54:27,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7365f32[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,784 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/7917cd30445048ec8bec409638dd2b4a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/7917cd30445048ec8bec409638dd2b4a 2023-03-26 11:54:27,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,787 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/7917cd30445048ec8bec409638dd2b4a, entries=150, sequenceid=497, filesize=11.9 K 2023-03-26 11:54:27,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 1279ms, sequenceid=497, compaction requested=true 2023-03-26 11:54:27,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:27,790 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:27,790 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:54:27,790 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:54:27,790 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:27,790 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,791 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:27,793 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 192098 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:54:27,793 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/A is initiating minor compaction (all files) 2023-03-26 11:54:27,793 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/A in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,793 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6180179ac282476f83868539f6f97b43, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a2c70d6bac834e648e0259266590ad7a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f64487f7b8d44c2fb20847ab227339ac, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/90668bbadc6f48fda564933de008c260, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/de5b100d32db4319b39b3f29b3881ccf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5286bdedb8214a3da0cc2c77a0724d55] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=187.6 K 2023-03-26 11:54:27,794 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:54:27,794 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/B is initiating minor compaction (all files) 2023-03-26 11:54:27,794 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/B in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,794 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/cf657b9a4fbc4de090c3de5962c503ef, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/60a532cb64264a2db86ce0a097bcd954, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/57633d8eb37243f7abff55e4aa0a3e37, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/2d45e6a6e13549f29dc2b914cfa405f0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/9ba6a6cb203f43519d6a7a4e6efe6ec2] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=72.0 K 2023-03-26 11:54:27,794 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 6180179ac282476f83868539f6f97b43, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679831663137 2023-03-26 11:54:27,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,795 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting cf657b9a4fbc4de090c3de5962c503ef, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679831663137 2023-03-26 11:54:27,795 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a2c70d6bac834e648e0259266590ad7a, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679831663434 2023-03-26 11:54:27,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,795 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 60a532cb64264a2db86ce0a097bcd954, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679831663434 2023-03-26 11:54:27,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,796 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f64487f7b8d44c2fb20847ab227339ac, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1679831663681 2023-03-26 11:54:27,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,796 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 57633d8eb37243f7abff55e4aa0a3e37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1679831663681 2023-03-26 11:54:27,796 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 90668bbadc6f48fda564933de008c260, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679831664192 2023-03-26 11:54:27,796 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 4cc9e3203f0d47b4bc1cc47b87a8fd8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679831664192 2023-03-26 11:54:27,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,797 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting de5b100d32db4319b39b3f29b3881ccf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1679831665373 2023-03-26 11:54:27,797 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2d45e6a6e13549f29dc2b914cfa405f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1679831665373 2023-03-26 11:54:27,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,798 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5286bdedb8214a3da0cc2c77a0724d55, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1679831666452 2023-03-26 11:54:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,798 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9ba6a6cb203f43519d6a7a4e6efe6ec2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1679831666499 2023-03-26 11:54:27,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,802 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,802 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:27,802 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,802 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,815 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#B#compaction#194 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:27,825 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#A#compaction#195 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:27,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:27,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-26 11:54:27,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:27,851 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:27,851 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:27,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:27,852 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:27,852 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:27,886 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:27,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831727881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,902 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:27,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:27,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:27,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:27,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:27,906 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:27,906 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,906 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:27,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:27,937 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,937 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,938 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,939 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:27,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,940 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:27,940 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:27,940 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,940 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:27,940 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:27,940 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:27,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:27,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,942 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,942 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:27,944 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:27,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b5f910a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:27,944 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:27,946 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:27,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:27,948 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:27,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:27,950 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:27,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:27,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:27,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:27,950 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:27,952 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,952 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,952 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:27,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:27,953 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:27,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,955 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:27,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:27,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,958 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:27,958 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:27,958 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,958 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,958 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:27,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:27,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:27,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:27,958 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:27,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:27,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:27,959 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:27,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:27,958 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:27,959 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:27,959 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,959 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:27,959 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:27,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:27,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831727988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,059 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,062 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,067 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,069 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,069 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,069 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,069 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,071 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29ba2894[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,073 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,076 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,078 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,078 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,080 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,085 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,086 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,086 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,186 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,189 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,190 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,190 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,195 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:28,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831728193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,197 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,197 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,198 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,198 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,198 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,198 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,199 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,199 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,199 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,199 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,205 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45fac64e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,207 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,207 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,208 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,208 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,208 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,210 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,210 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,212 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,212 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,214 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,214 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,214 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,214 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,216 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,219 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,219 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,219 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:28,219 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,219 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,220 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,229 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/77106129ee4a4885acb7756c94f347e9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/77106129ee4a4885acb7756c94f347e9 2023-03-26 11:54:28,233 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/B of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into 77106129ee4a4885acb7756c94f347e9(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:28,233 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:28,233 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/B, priority=10, startTime=1679831667790; duration=0sec 2023-03-26 11:54:28,233 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:28,233 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:54:28,235 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73730 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:54:28,236 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 2c5753ba51ccea59bc0c70ae6d9ee6c2/C is initiating minor compaction (all files) 2023-03-26 11:54:28,236 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 2c5753ba51ccea59bc0c70ae6d9ee6c2/C in TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,236 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/be942c29768a43b796fb365d3682c331, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/aeb9119e1b1d4dd79d80eeba74a24979, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/d228456721504044a1fe658ab78d1731, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/bae97e3faf5040a5b2dd767b920f88ee, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3641ce19d46b424ebb69c71338a8759f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/7917cd30445048ec8bec409638dd2b4a] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp, totalSize=72.0 K 2023-03-26 11:54:28,236 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting be942c29768a43b796fb365d3682c331, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=390, earliestPutTs=1679831663137 2023-03-26 11:54:28,237 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting aeb9119e1b1d4dd79d80eeba74a24979, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679831663434 2023-03-26 11:54:28,237 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d228456721504044a1fe658ab78d1731, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=440, earliestPutTs=1679831663681 2023-03-26 11:54:28,237 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bae97e3faf5040a5b2dd767b920f88ee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679831664192 2023-03-26 11:54:28,238 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3641ce19d46b424ebb69c71338a8759f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=487, earliestPutTs=1679831665373 2023-03-26 11:54:28,238 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 7917cd30445048ec8bec409638dd2b4a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1679831666499 2023-03-26 11:54:28,244 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/88e37b27eaa9460a951fb6c3d6f94729 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/88e37b27eaa9460a951fb6c3d6f94729 2023-03-26 11:54:28,249 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/A of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into 88e37b27eaa9460a951fb6c3d6f94729(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:28,249 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:28,250 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/A, priority=10, startTime=1679831667790; duration=0sec 2023-03-26 11:54:28,250 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:28,258 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 2c5753ba51ccea59bc0c70ae6d9ee6c2#C#compaction#197 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:28,272 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267557aea8979d4f98ae098a40222afec9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267557aea8979d4f98ae098a40222afec9 2023-03-26 11:54:28,296 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/cf293080205a4a3db5a0e97d08271fab as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cf293080205a4a3db5a0e97d08271fab 2023-03-26 11:54:28,302 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 2c5753ba51ccea59bc0c70ae6d9ee6c2/C of 2c5753ba51ccea59bc0c70ae6d9ee6c2 into cf293080205a4a3db5a0e97d08271fab(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:28,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:28,302 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2., storeName=2c5753ba51ccea59bc0c70ae6d9ee6c2/C, priority=10, startTime=1679831667791; duration=0sec 2023-03-26 11:54:28,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:28,320 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,322 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,323 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,323 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,323 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,326 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,327 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,327 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,327 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,329 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7080ce0c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,334 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,341 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,341 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,341 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,341 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,341 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,352 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,353 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,355 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,355 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,355 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,355 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,358 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,358 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,458 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,461 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,463 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,463 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,463 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,463 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,463 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,463 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,465 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,465 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,465 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,466 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,467 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,472 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,472 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6740c6d7[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,493 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,497 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,497 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:28,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831728496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,500 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,500 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,500 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,503 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:28,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,509 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,509 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,609 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,613 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,614 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,614 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,615 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,617 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,617 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,617 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,617 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,620 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,622 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fc6ac23[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,622 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,622 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,623 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,626 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,628 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,628 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,628 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,631 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,634 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,634 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,635 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,635 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,715 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=534, memsize=47.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/4fc2c7fb4f814ff58925e89cf4eef19f 2023-03-26 11:54:28,727 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/8d370c92cc4f4e68951c705447d31cbd 2023-03-26 11:54:28,733 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:28,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57118 deadline: 1679831728731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,735 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,736 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:28,736 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57134 deadline: 1679831728733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,739 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,740 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,740 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,741 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,742 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,750 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,750 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,750 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,750 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,750 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,750 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,750 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,753 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,754 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@c9dc123[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,755 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,755 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,757 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,759 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,759 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,759 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,761 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,761 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,761 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,761 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,761 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,765 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,769 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,769 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,769 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,870 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:28,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:28,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:28,873 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:28,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:28,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:28,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:28,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,876 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:28,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:28,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:28,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:28,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:28,878 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,878 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:28,878 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:28,878 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:28,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:28,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:28,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:28,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:28,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,880 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:28,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a9b9e12[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:28,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:28,882 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:28,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:28,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:28,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:28,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:28,890 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:28,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:28,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:28,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:28,890 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:28,892 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,892 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,892 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:28,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:28,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:28,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:28,896 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:28,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:28,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:28,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:28,896 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:28,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:28,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:28,896 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:28,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:28,997 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:28,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:28,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:29,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:29,000 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:29,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:29,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:29,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:29,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:29,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:29,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:29,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:29,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:57074 deadline: 1679831729002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,004 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:29,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:29,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:29,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:29,005 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:29,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:29,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:29,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b0c89e8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:29,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:29,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:29,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:29,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:29,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:29,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:29,012 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:29,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:29,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:29,029 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:29,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:29,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:29,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:29,029 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:29,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:29,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:29,033 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,033 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:29,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:29,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:29,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,037 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:29,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:29,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:29,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:29,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:29,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:29,043 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:29,043 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:29,145 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:29,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:29,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:29,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:29,151 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:29,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:29,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:29,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:29,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,155 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=534 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/e5ef24ba8e3c4e0bacc11e5524111bd3 2023-03-26 11:54:29,157 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:29,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:29,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,158 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:29,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,159 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:29,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:29,164 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:29,164 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:29,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:29,168 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/4fc2c7fb4f814ff58925e89cf4eef19f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/4fc2c7fb4f814ff58925e89cf4eef19f 2023-03-26 11:54:29,179 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/4fc2c7fb4f814ff58925e89cf4eef19f, entries=200, sequenceid=534, filesize=32.3 K 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:29,180 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/8d370c92cc4f4e68951c705447d31cbd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/8d370c92cc4f4e68951c705447d31cbd 2023-03-26 11:54:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:29,180 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:29,180 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:29,183 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:29,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:29,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,184 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:29,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3101991e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:29,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:29,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:29,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:29,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:29,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:29,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,189 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/8d370c92cc4f4e68951c705447d31cbd, entries=150, sequenceid=534, filesize=11.9 K 2023-03-26 11:54:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:29,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:29,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:29,190 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/e5ef24ba8e3c4e0bacc11e5524111bd3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/e5ef24ba8e3c4e0bacc11e5524111bd3 2023-03-26 11:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:29,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:29,192 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:29,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:29,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:29,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:29,192 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:29,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:29,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:29,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:29,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/e5ef24ba8e3c4e0bacc11e5524111bd3, entries=150, sequenceid=534, filesize=11.9 K 2023-03-26 11:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:29,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,196 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 1345ms, sequenceid=534, compaction requested=false 2023-03-26 11:54:29,196 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:29,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:29,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:29,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:29,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:29,201 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:29,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:29,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:29,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:29,201 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:29,201 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:29,201 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,301 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:29,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:29,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:29,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:29,305 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,307 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:29,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:29,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:29,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:29,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:29,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:29,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:29,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:29,312 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:29,312 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:29,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:29,312 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:29,312 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. started... 2023-03-26 11:54:29,313 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 11:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:29,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:29,347 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326785f210f7d1d4cd88c825a5c9457aa9b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326785f210f7d1d4cd88c825a5c9457aa9b 2023-03-26 11:54:29,763 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=546, memsize=13.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/9a23a02839d64ca8a580e8ea53648078 2023-03-26 11:54:29,804 DEBUG [Thread-959] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:53955 2023-03-26 11:54:29,805 DEBUG [Thread-959] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:29,805 DEBUG [Thread-957] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:53955 2023-03-26 11:54:29,805 DEBUG [Thread-957] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:29,820 DEBUG [Thread-961] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:53955 2023-03-26 11:54:29,820 DEBUG [Thread-961] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:29,820 DEBUG [Thread-955] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:53955 2023-03-26 11:54:29,820 DEBUG [Thread-955] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:30,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:30,013 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. as already flushing 2023-03-26 11:54:30,013 DEBUG [Thread-948] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:53955 2023-03-26 11:54:30,013 DEBUG [Thread-948] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:30,177 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/baa0ac92594947f0a1748dd19a63f667 2023-03-26 11:54:30,421 DEBUG [Thread-950] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:53955 2023-03-26 11:54:30,421 DEBUG [Thread-950] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:30,586 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=546 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/a057ba5b2a8a41e78251279a81d46fc9 2023-03-26 11:54:30,590 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/9a23a02839d64ca8a580e8ea53648078 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a23a02839d64ca8a580e8ea53648078 2023-03-26 11:54:30,593 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a23a02839d64ca8a580e8ea53648078, entries=100, sequenceid=546, filesize=18.6 K 2023-03-26 11:54:30,594 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/baa0ac92594947f0a1748dd19a63f667 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/baa0ac92594947f0a1748dd19a63f667 2023-03-26 11:54:30,597 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/baa0ac92594947f0a1748dd19a63f667, entries=100, sequenceid=546, filesize=9.5 K 2023-03-26 11:54:30,597 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/a057ba5b2a8a41e78251279a81d46fc9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/a057ba5b2a8a41e78251279a81d46fc9 2023-03-26 11:54:30,600 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/a057ba5b2a8a41e78251279a81d46fc9, entries=100, sequenceid=546, filesize=9.5 K 2023-03-26 11:54:30,601 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=13.42 KB/13740 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 1289ms, sequenceid=546, compaction requested=true 2023-03-26 11:54:30,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:30,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:30,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:30,604 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,604 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:30,604 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,604 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:30,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:30,605 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:30,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5377c2c4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:30,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:30,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:30,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:30,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:30,607 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,607 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:30,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:30,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:30,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:30,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:30,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:30,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:30,613 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:30,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:30,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:30,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:30,613 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:30,614 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,614 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,614 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:30,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:30,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:30,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:30,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:30,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:30,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:30,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:30,618 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:30,618 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:30,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:30,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:30,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:30,618 DEBUG [Thread-954] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:30,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:30,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:30,618 DEBUG [Thread-954] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:30,719 DEBUG [Thread-954] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:30,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:30,744 DEBUG [Thread-944] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34135972 to 127.0.0.1:53955 2023-03-26 11:54:30,744 DEBUG [Thread-944] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:30,752 DEBUG [Thread-946] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:53955 2023-03-26 11:54:30,752 DEBUG [Thread-946] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:32,188 DEBUG [Thread-952] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:53955 2023-03-26 11:54:32,188 DEBUG [Thread-952] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:32,188 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 11:54:32,188 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 124 2023-03-26 11:54:32,188 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 154 2023-03-26 11:54:32,188 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 145 2023-03-26 11:54:32,188 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 16 2023-03-26 11:54:32,188 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 27 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 3773 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 3781 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1660 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 4980 rows 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1650 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 4950 rows 2023-03-26 11:54:32,189 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:54:32,189 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52274650 to 127.0.0.1:53955 2023-03-26 11:54:32,189 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:32,190 INFO [Listener at localhost/46409] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 11:54:32,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 11:54:32,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:32,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 11:54:32,195 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831672195"}]},"ts":"1679831672195"} 2023-03-26 11:54:32,196 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 11:54:32,198 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 11:54:32,198 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, UNASSIGN}] 2023-03-26 11:54:32,200 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, UNASSIGN 2023-03-26 11:54:32,200 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:32,200 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831672200"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831672200"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831672200"}]},"ts":"1679831672200"} 2023-03-26 11:54:32,201 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:32,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 11:54:32,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 2c5753ba51ccea59bc0c70ae6d9ee6c2, disabling compactions & flushes 2023-03-26 11:54:32,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. after waiting 0 ms 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:32,353 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 2c5753ba51ccea59bc0c70ae6d9ee6c2 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=A 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:32,353 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=B 2023-03-26 11:54:32,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:32,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 2c5753ba51ccea59bc0c70ae6d9ee6c2, store=C 2023-03-26 11:54:32,354 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:32,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 11:54:32,765 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032644c2f6cbb5bb4a26a66794767e82139b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032644c2f6cbb5bb4a26a66794767e82139b 2023-03-26 11:54:32,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 11:54:33,170 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=554, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/c2065ee4f115448fb0785ba52e865346 2023-03-26 11:54:33,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 11:54:33,579 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/3a2dbdd3e2974a31825b59da7809cb91 2023-03-26 11:54:33,988 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=554 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/6d549e29f9ff4f6cb91a938e30cefe0c 2023-03-26 11:54:33,992 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/A/c2065ee4f115448fb0785ba52e865346 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/c2065ee4f115448fb0785ba52e865346 2023-03-26 11:54:33,995 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/c2065ee4f115448fb0785ba52e865346, entries=150, sequenceid=554, filesize=25.4 K 2023-03-26 11:54:33,996 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/B/3a2dbdd3e2974a31825b59da7809cb91 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/3a2dbdd3e2974a31825b59da7809cb91 2023-03-26 11:54:33,999 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/3a2dbdd3e2974a31825b59da7809cb91, entries=150, sequenceid=554, filesize=11.9 K 2023-03-26 11:54:34,000 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/.tmp/C/6d549e29f9ff4f6cb91a938e30cefe0c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d549e29f9ff4f6cb91a938e30cefe0c 2023-03-26 11:54:34,003 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d549e29f9ff4f6cb91a938e30cefe0c, entries=150, sequenceid=554, filesize=11.9 K 2023-03-26 11:54:34,004 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 2c5753ba51ccea59bc0c70ae6d9ee6c2 in 1651ms, sequenceid=554, compaction requested=true 2023-03-26 11:54:34,024 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce90e8047e95489d9ce73a997a6df16d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/d948bec8e0fb4917838b72062f79c986, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a6a0aa3f0b634e5b9170525360609acf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/1dae363c6c6f4548984524e9534bec94, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/2179ab0afcab4d8bad105a109d407b87, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5937e5996e3d4afb9b2f8b8dd6280ebe, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5befde2e8966417aa3a8e5c1557ac24d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/082a7e0cab2e47cfaaad738ffdb67864, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce210e64ff6746fda5db149df4518e22, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5224a3126ffc4e0396c225603300ae1c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/aafedb47ba1640838926fdcdf9966ea4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/b2054a2e12ab448dac5d2cee4c1ebc7c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f2472432617c410c9c54c478d7a8f46c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/65986d747815493a9180d5771ba3cd98, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a20a02221f349c7832324d5ede537af, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/74d758bd32fc40f6a94cb2199984fe52, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6b2de7b5abcc444a8cdcd9a2e7d6f458, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/36b1881a477b49218447fc23e2691565, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6180179ac282476f83868539f6f97b43, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a2c70d6bac834e648e0259266590ad7a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f64487f7b8d44c2fb20847ab227339ac, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/90668bbadc6f48fda564933de008c260, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/de5b100d32db4319b39b3f29b3881ccf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5286bdedb8214a3da0cc2c77a0724d55] to archive 2023-03-26 11:54:34,025 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:54:34,027 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce90e8047e95489d9ce73a997a6df16d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce90e8047e95489d9ce73a997a6df16d 2023-03-26 11:54:34,028 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/d948bec8e0fb4917838b72062f79c986 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/d948bec8e0fb4917838b72062f79c986 2023-03-26 11:54:34,029 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a6a0aa3f0b634e5b9170525360609acf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a6a0aa3f0b634e5b9170525360609acf 2023-03-26 11:54:34,030 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/1dae363c6c6f4548984524e9534bec94 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/1dae363c6c6f4548984524e9534bec94 2023-03-26 11:54:34,031 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/2179ab0afcab4d8bad105a109d407b87 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/2179ab0afcab4d8bad105a109d407b87 2023-03-26 11:54:34,032 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5937e5996e3d4afb9b2f8b8dd6280ebe to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5937e5996e3d4afb9b2f8b8dd6280ebe 2023-03-26 11:54:34,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5befde2e8966417aa3a8e5c1557ac24d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5befde2e8966417aa3a8e5c1557ac24d 2023-03-26 11:54:34,033 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/082a7e0cab2e47cfaaad738ffdb67864 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/082a7e0cab2e47cfaaad738ffdb67864 2023-03-26 11:54:34,034 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce210e64ff6746fda5db149df4518e22 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/ce210e64ff6746fda5db149df4518e22 2023-03-26 11:54:34,035 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5224a3126ffc4e0396c225603300ae1c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5224a3126ffc4e0396c225603300ae1c 2023-03-26 11:54:34,036 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/aafedb47ba1640838926fdcdf9966ea4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/aafedb47ba1640838926fdcdf9966ea4 2023-03-26 11:54:34,037 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/b2054a2e12ab448dac5d2cee4c1ebc7c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/b2054a2e12ab448dac5d2cee4c1ebc7c 2023-03-26 11:54:34,038 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f2472432617c410c9c54c478d7a8f46c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f2472432617c410c9c54c478d7a8f46c 2023-03-26 11:54:34,039 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/65986d747815493a9180d5771ba3cd98 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/65986d747815493a9180d5771ba3cd98 2023-03-26 11:54:34,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a20a02221f349c7832324d5ede537af to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a20a02221f349c7832324d5ede537af 2023-03-26 11:54:34,040 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/74d758bd32fc40f6a94cb2199984fe52 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/74d758bd32fc40f6a94cb2199984fe52 2023-03-26 11:54:34,041 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6b2de7b5abcc444a8cdcd9a2e7d6f458 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6b2de7b5abcc444a8cdcd9a2e7d6f458 2023-03-26 11:54:34,042 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/36b1881a477b49218447fc23e2691565 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/36b1881a477b49218447fc23e2691565 2023-03-26 11:54:34,043 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6180179ac282476f83868539f6f97b43 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/6180179ac282476f83868539f6f97b43 2023-03-26 11:54:34,045 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a2c70d6bac834e648e0259266590ad7a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/a2c70d6bac834e648e0259266590ad7a 2023-03-26 11:54:34,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f64487f7b8d44c2fb20847ab227339ac to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/f64487f7b8d44c2fb20847ab227339ac 2023-03-26 11:54:34,046 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/90668bbadc6f48fda564933de008c260 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/90668bbadc6f48fda564933de008c260 2023-03-26 11:54:34,047 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/de5b100d32db4319b39b3f29b3881ccf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/de5b100d32db4319b39b3f29b3881ccf 2023-03-26 11:54:34,049 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5286bdedb8214a3da0cc2c77a0724d55 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/5286bdedb8214a3da0cc2c77a0724d55 2023-03-26 11:54:34,052 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/795ff571ef934867a3ae7f74ae63eaae, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e8bf0ef727214d9582562531dbbad12a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ecd43671e2f94180a86aeaf9fbbf8fca, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ae92a7b6b72c40b8a3b0395c6d3d39a6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/45bdcae1768e4ea4a82b4069762094c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/519895e7d71147cda118dbbff315d9bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/54798523e6164c35a20903d2c3503919, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e48c03fcdca34c7b8eb17b01f565c6d5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e055344ca8f241cc8a545c573e8f98c0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b9b834844f5f490f852893f7a7885df2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/0c1f0a6941154d509d3df1004bd33521, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c0717a3091094a58b2668ade2bf60aac, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c28e17fc90bd4d088dd4cd4c466c3b17, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/dea1aab82e364b79979048fe1b7e12b6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ed625797b4ad4114a27ea6c6b0848948, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/74ff946c56534f73bcaff9c2bc11672c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b3fde9723d9f4817b9e2126669a01562, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/cf657b9a4fbc4de090c3de5962c503ef, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/5a81ec54a6724812bf1f379cd2925093, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/60a532cb64264a2db86ce0a097bcd954, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/57633d8eb37243f7abff55e4aa0a3e37, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/2d45e6a6e13549f29dc2b914cfa405f0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/9ba6a6cb203f43519d6a7a4e6efe6ec2] to archive 2023-03-26 11:54:34,053 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:54:34,055 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/795ff571ef934867a3ae7f74ae63eaae to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/795ff571ef934867a3ae7f74ae63eaae 2023-03-26 11:54:34,056 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e8bf0ef727214d9582562531dbbad12a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e8bf0ef727214d9582562531dbbad12a 2023-03-26 11:54:34,057 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ecd43671e2f94180a86aeaf9fbbf8fca to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ecd43671e2f94180a86aeaf9fbbf8fca 2023-03-26 11:54:34,058 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ae92a7b6b72c40b8a3b0395c6d3d39a6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ae92a7b6b72c40b8a3b0395c6d3d39a6 2023-03-26 11:54:34,059 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/45bdcae1768e4ea4a82b4069762094c1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/45bdcae1768e4ea4a82b4069762094c1 2023-03-26 11:54:34,061 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/519895e7d71147cda118dbbff315d9bf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/519895e7d71147cda118dbbff315d9bf 2023-03-26 11:54:34,062 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/54798523e6164c35a20903d2c3503919 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/54798523e6164c35a20903d2c3503919 2023-03-26 11:54:34,063 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e48c03fcdca34c7b8eb17b01f565c6d5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e48c03fcdca34c7b8eb17b01f565c6d5 2023-03-26 11:54:34,064 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e055344ca8f241cc8a545c573e8f98c0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/e055344ca8f241cc8a545c573e8f98c0 2023-03-26 11:54:34,065 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b9b834844f5f490f852893f7a7885df2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b9b834844f5f490f852893f7a7885df2 2023-03-26 11:54:34,066 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/0c1f0a6941154d509d3df1004bd33521 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/0c1f0a6941154d509d3df1004bd33521 2023-03-26 11:54:34,068 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c0717a3091094a58b2668ade2bf60aac to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c0717a3091094a58b2668ade2bf60aac 2023-03-26 11:54:34,069 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c28e17fc90bd4d088dd4cd4c466c3b17 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/c28e17fc90bd4d088dd4cd4c466c3b17 2023-03-26 11:54:34,070 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/dea1aab82e364b79979048fe1b7e12b6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/dea1aab82e364b79979048fe1b7e12b6 2023-03-26 11:54:34,071 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ed625797b4ad4114a27ea6c6b0848948 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/ed625797b4ad4114a27ea6c6b0848948 2023-03-26 11:54:34,072 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/74ff946c56534f73bcaff9c2bc11672c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/74ff946c56534f73bcaff9c2bc11672c 2023-03-26 11:54:34,073 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b3fde9723d9f4817b9e2126669a01562 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/b3fde9723d9f4817b9e2126669a01562 2023-03-26 11:54:34,074 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/cf657b9a4fbc4de090c3de5962c503ef to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/cf657b9a4fbc4de090c3de5962c503ef 2023-03-26 11:54:34,075 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/5a81ec54a6724812bf1f379cd2925093 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/5a81ec54a6724812bf1f379cd2925093 2023-03-26 11:54:34,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/60a532cb64264a2db86ce0a097bcd954 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/60a532cb64264a2db86ce0a097bcd954 2023-03-26 11:54:34,077 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/57633d8eb37243f7abff55e4aa0a3e37 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/57633d8eb37243f7abff55e4aa0a3e37 2023-03-26 11:54:34,078 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/4cc9e3203f0d47b4bc1cc47b87a8fd8b 2023-03-26 11:54:34,079 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/2d45e6a6e13549f29dc2b914cfa405f0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/2d45e6a6e13549f29dc2b914cfa405f0 2023-03-26 11:54:34,080 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/9ba6a6cb203f43519d6a7a4e6efe6ec2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/9ba6a6cb203f43519d6a7a4e6efe6ec2 2023-03-26 11:54:34,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c99af99d356c4034b55912a1ff7524f3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/1b4debb80f2d4cb2b86c0c69d1ccaf67, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/43a269e6fcfd4f0491aef003c7dfd2a8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6e23c91fc5dc460298e846227aee16f5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/ff9755454a9949979f6cde939b3746af, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d778a2fbbbb41958415fb10f131e1b0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/dd8bcebdd2b94cb18aedb09238ad63c7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c2d64e564688479a958841b90bf54279, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3f2e91edd77d4f1c93a8555f4a103b5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/f336c8428c414a888a520f28543ce029, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/33275713e30a463a8a42384fcf7dd575, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/4cf7a32af0c943bb9f8a1f12b4659c4d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/91352d576704495a8de3fd4a1e14fbe2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c91b6f08daa44be69d4ce3f4c2f1908a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cbe265f7df104f89a5dbecf5a095050f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/249bbafcdc7448f6b28eadfd9353d07f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/b2fe06b1f1b942ad8815f05385e8de8c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/be942c29768a43b796fb365d3682c331, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/55f98d7f5aa84b4490b23dfdb37d260c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/aeb9119e1b1d4dd79d80eeba74a24979, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/d228456721504044a1fe658ab78d1731, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/bae97e3faf5040a5b2dd767b920f88ee, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3641ce19d46b424ebb69c71338a8759f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/7917cd30445048ec8bec409638dd2b4a] to archive 2023-03-26 11:54:34,083 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:54:34,085 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c99af99d356c4034b55912a1ff7524f3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c99af99d356c4034b55912a1ff7524f3 2023-03-26 11:54:34,086 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/1b4debb80f2d4cb2b86c0c69d1ccaf67 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/1b4debb80f2d4cb2b86c0c69d1ccaf67 2023-03-26 11:54:34,087 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/43a269e6fcfd4f0491aef003c7dfd2a8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/43a269e6fcfd4f0491aef003c7dfd2a8 2023-03-26 11:54:34,088 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6e23c91fc5dc460298e846227aee16f5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6e23c91fc5dc460298e846227aee16f5 2023-03-26 11:54:34,089 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/ff9755454a9949979f6cde939b3746af to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/ff9755454a9949979f6cde939b3746af 2023-03-26 11:54:34,090 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d778a2fbbbb41958415fb10f131e1b0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d778a2fbbbb41958415fb10f131e1b0 2023-03-26 11:54:34,091 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/dd8bcebdd2b94cb18aedb09238ad63c7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/dd8bcebdd2b94cb18aedb09238ad63c7 2023-03-26 11:54:34,092 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c2d64e564688479a958841b90bf54279 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c2d64e564688479a958841b90bf54279 2023-03-26 11:54:34,093 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3f2e91edd77d4f1c93a8555f4a103b5f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3f2e91edd77d4f1c93a8555f4a103b5f 2023-03-26 11:54:34,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/f336c8428c414a888a520f28543ce029 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/f336c8428c414a888a520f28543ce029 2023-03-26 11:54:34,094 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/33275713e30a463a8a42384fcf7dd575 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/33275713e30a463a8a42384fcf7dd575 2023-03-26 11:54:34,095 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/4cf7a32af0c943bb9f8a1f12b4659c4d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/4cf7a32af0c943bb9f8a1f12b4659c4d 2023-03-26 11:54:34,096 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/91352d576704495a8de3fd4a1e14fbe2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/91352d576704495a8de3fd4a1e14fbe2 2023-03-26 11:54:34,097 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c91b6f08daa44be69d4ce3f4c2f1908a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/c91b6f08daa44be69d4ce3f4c2f1908a 2023-03-26 11:54:34,098 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cbe265f7df104f89a5dbecf5a095050f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cbe265f7df104f89a5dbecf5a095050f 2023-03-26 11:54:34,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/249bbafcdc7448f6b28eadfd9353d07f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/249bbafcdc7448f6b28eadfd9353d07f 2023-03-26 11:54:34,099 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/b2fe06b1f1b942ad8815f05385e8de8c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/b2fe06b1f1b942ad8815f05385e8de8c 2023-03-26 11:54:34,100 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/be942c29768a43b796fb365d3682c331 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/be942c29768a43b796fb365d3682c331 2023-03-26 11:54:34,101 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/55f98d7f5aa84b4490b23dfdb37d260c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/55f98d7f5aa84b4490b23dfdb37d260c 2023-03-26 11:54:34,102 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/aeb9119e1b1d4dd79d80eeba74a24979 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/aeb9119e1b1d4dd79d80eeba74a24979 2023-03-26 11:54:34,103 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/d228456721504044a1fe658ab78d1731 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/d228456721504044a1fe658ab78d1731 2023-03-26 11:54:34,104 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/bae97e3faf5040a5b2dd767b920f88ee to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/bae97e3faf5040a5b2dd767b920f88ee 2023-03-26 11:54:34,105 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3641ce19d46b424ebb69c71338a8759f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/3641ce19d46b424ebb69c71338a8759f 2023-03-26 11:54:34,106 DEBUG [StoreCloser-TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/7917cd30445048ec8bec409638dd2b4a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/7917cd30445048ec8bec409638dd2b4a 2023-03-26 11:54:34,110 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/recovered.edits/557.seqid, newMaxSeqId=557, maxSeqId=4 2023-03-26 11:54:34,111 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2. 2023-03-26 11:54:34,111 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 2c5753ba51ccea59bc0c70ae6d9ee6c2: 2023-03-26 11:54:34,112 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:34,113 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=2c5753ba51ccea59bc0c70ae6d9ee6c2, regionState=CLOSED 2023-03-26 11:54:34,113 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831674113"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831674113"}]},"ts":"1679831674113"} 2023-03-26 11:54:34,115 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-03-26 11:54:34,115 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure 2c5753ba51ccea59bc0c70ae6d9ee6c2, server=jenkins-hbase4.apache.org,38971,1679831615531 in 1.9130 sec 2023-03-26 11:54:34,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-03-26 11:54:34,116 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=2c5753ba51ccea59bc0c70ae6d9ee6c2, UNASSIGN in 1.9170 sec 2023-03-26 11:54:34,117 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831674117"}]},"ts":"1679831674117"} 2023-03-26 11:54:34,117 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 11:54:34,120 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 11:54:34,121 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9290 sec 2023-03-26 11:54:34,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-03-26 11:54:34,299 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-03-26 11:54:34,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 11:54:34,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,302 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-26 11:54:34,302 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,307 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:34,309 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/recovered.edits] 2023-03-26 11:54:34,311 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/4fc2c7fb4f814ff58925e89cf4eef19f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/4fc2c7fb4f814ff58925e89cf4eef19f 2023-03-26 11:54:34,312 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/88e37b27eaa9460a951fb6c3d6f94729 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/88e37b27eaa9460a951fb6c3d6f94729 2023-03-26 11:54:34,313 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a23a02839d64ca8a580e8ea53648078 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/9a23a02839d64ca8a580e8ea53648078 2023-03-26 11:54:34,314 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/c2065ee4f115448fb0785ba52e865346 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/A/c2065ee4f115448fb0785ba52e865346 2023-03-26 11:54:34,316 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/3a2dbdd3e2974a31825b59da7809cb91 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/3a2dbdd3e2974a31825b59da7809cb91 2023-03-26 11:54:34,317 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/77106129ee4a4885acb7756c94f347e9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/77106129ee4a4885acb7756c94f347e9 2023-03-26 11:54:34,318 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/8d370c92cc4f4e68951c705447d31cbd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/8d370c92cc4f4e68951c705447d31cbd 2023-03-26 11:54:34,320 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/baa0ac92594947f0a1748dd19a63f667 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/B/baa0ac92594947f0a1748dd19a63f667 2023-03-26 11:54:34,322 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d549e29f9ff4f6cb91a938e30cefe0c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/6d549e29f9ff4f6cb91a938e30cefe0c 2023-03-26 11:54:34,323 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/a057ba5b2a8a41e78251279a81d46fc9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/a057ba5b2a8a41e78251279a81d46fc9 2023-03-26 11:54:34,324 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cf293080205a4a3db5a0e97d08271fab to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/cf293080205a4a3db5a0e97d08271fab 2023-03-26 11:54:34,325 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/e5ef24ba8e3c4e0bacc11e5524111bd3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/C/e5ef24ba8e3c4e0bacc11e5524111bd3 2023-03-26 11:54:34,328 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/recovered.edits/557.seqid to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2/recovered.edits/557.seqid 2023-03-26 11:54:34,328 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/2c5753ba51ccea59bc0c70ae6d9ee6c2 2023-03-26 11:54:34,328 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:54:34,329 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 11:54:34,329 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-26 11:54:34,332 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260552bfee0c90416d92d04dd7c993dc67 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303260552bfee0c90416d92d04dd7c993dc67 2023-03-26 11:54:34,333 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032607e887f9f3f54c3687f2af86e0eb700a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032607e887f9f3f54c3687f2af86e0eb700a 2023-03-26 11:54:34,334 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032621256f809c444a698623bb4db14cb344 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032621256f809c444a698623bb4db14cb344 2023-03-26 11:54:34,335 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032623a34e499b0a43c6ad6072284721f9fa to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032623a34e499b0a43c6ad6072284721f9fa 2023-03-26 11:54:34,336 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262d4c48a79a3b4c3185c2444735975cf9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262d4c48a79a3b4c3185c2444735975cf9 2023-03-26 11:54:34,337 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263647cf1eea994114af7d4fcdfd0484bc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263647cf1eea994114af7d4fcdfd0484bc 2023-03-26 11:54:34,338 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263e669cc91f8d400e9b46a41abdd17601 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263e669cc91f8d400e9b46a41abdd17601 2023-03-26 11:54:34,339 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032644c2f6cbb5bb4a26a66794767e82139b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032644c2f6cbb5bb4a26a66794767e82139b 2023-03-26 11:54:34,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264f18bdf4069f4f9094ea1a4693339afe to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264f18bdf4069f4f9094ea1a4693339afe 2023-03-26 11:54:34,340 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267557aea8979d4f98ae098a40222afec9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267557aea8979d4f98ae098a40222afec9 2023-03-26 11:54:34,342 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326785f210f7d1d4cd88c825a5c9457aa9b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326785f210f7d1d4cd88c825a5c9457aa9b 2023-03-26 11:54:34,343 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032678761bedd9464e59a30a67fe76f4cb95 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032678761bedd9464e59a30a67fe76f4cb95 2023-03-26 11:54:34,344 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682722cb793a24460bf0b9069fc12e2c0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682722cb793a24460bf0b9069fc12e2c0 2023-03-26 11:54:34,345 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682e005f163b941cba782067163ee1573 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682e005f163b941cba782067163ee1573 2023-03-26 11:54:34,346 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268b9db3d7e9384015afd4ce30b8994560 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303268b9db3d7e9384015afd4ce30b8994560 2023-03-26 11:54:34,347 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269a4dbcb97705402da0d624331a825245 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269a4dbcb97705402da0d624331a825245 2023-03-26 11:54:34,348 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326adb2d866645a45b9b23552b97999e1f3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326adb2d866645a45b9b23552b97999e1f3 2023-03-26 11:54:34,349 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326bee46692a0aa4fe7bf206346b769d489 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326bee46692a0aa4fe7bf206346b769d489 2023-03-26 11:54:34,350 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d3774e38990247daa981f94b05c1f46f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326d3774e38990247daa981f94b05c1f46f 2023-03-26 11:54:34,352 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326def4bb54df6742e2a2fa5df7847a89f0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326def4bb54df6742e2a2fa5df7847a89f0 2023-03-26 11:54:34,353 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326eaa087bf8dc54bd88aa97ace71ec7672 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326eaa087bf8dc54bd88aa97ace71ec7672 2023-03-26 11:54:34,354 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f104dd87de8644a0a9d4765ed934c49f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f104dd87de8644a0a9d4765ed934c49f 2023-03-26 11:54:34,355 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f831f336f4424ffaa3337d17582967a9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f831f336f4424ffaa3337d17582967a9 2023-03-26 11:54:34,356 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326fcb88aa4d12e45e68002828cc7d2d031 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326fcb88aa4d12e45e68002828cc7d2d031 2023-03-26 11:54:34,357 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 11:54:34,359 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,360 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 11:54:34,362 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 11:54:34,363 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,363 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 11:54:34,363 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679831674363"}]},"ts":"9223372036854775807"} 2023-03-26 11:54:34,365 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 11:54:34,365 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 2c5753ba51ccea59bc0c70ae6d9ee6c2, NAME => 'TestAcidGuarantees,,1679831647680.2c5753ba51ccea59bc0c70ae6d9ee6c2.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 11:54:34,365 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 11:54:34,365 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679831674365"}]},"ts":"9223372036854775807"} 2023-03-26 11:54:34,367 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 11:54:34,369 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,370 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 69 msec 2023-03-26 11:54:34,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-03-26 11:54:34,403 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-03-26 11:54:34,415 INFO [Listener at localhost/46409] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=234 (was 234), OpenFileDescriptor=583 (was 560) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=404 (was 403) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=8905 (was 9046) 2023-03-26 11:54:34,427 INFO [Listener at localhost/46409] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=234, OpenFileDescriptor=583, MaxFileDescriptor=60000, SystemLoadAverage=404, ProcessCount=169, AvailableMemoryMB=8905 2023-03-26 11:54:34,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:54:34,429 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:54:34,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:34,431 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:54:34,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-03-26 11:54:34,432 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:54:34,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 11:54:34,433 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:34,433 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da empty. 2023-03-26 11:54:34,434 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:34,434 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:54:34,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 11:54:34,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 11:54:34,841 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:54:34,843 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 70cba8f2e9bfa903d814db3e7b9776da, NAME => 'TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:54:35,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 11:54:35,248 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:54:35,249 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 70cba8f2e9bfa903d814db3e7b9776da, disabling compactions & flushes 2023-03-26 11:54:35,249 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,249 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,249 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. after waiting 0 ms 2023-03-26 11:54:35,249 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,249 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,249 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:35,251 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:54:35,252 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831675252"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831675252"}]},"ts":"1679831675252"} 2023-03-26 11:54:35,253 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:54:35,253 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:54:35,254 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831675253"}]},"ts":"1679831675253"} 2023-03-26 11:54:35,254 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 11:54:35,258 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, ASSIGN}] 2023-03-26 11:54:35,260 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, ASSIGN 2023-03-26 11:54:35,260 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:54:35,411 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=70cba8f2e9bfa903d814db3e7b9776da, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:35,411 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831675411"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831675411"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831675411"}]},"ts":"1679831675411"} 2023-03-26 11:54:35,413 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure 70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:35,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 11:54:35,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 70cba8f2e9bfa903d814db3e7b9776da, NAME => 'TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:54:35,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:54:35,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,568 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,569 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,570 DEBUG [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A 2023-03-26 11:54:35,570 DEBUG [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A 2023-03-26 11:54:35,571 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:35,571 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70cba8f2e9bfa903d814db3e7b9776da columnFamilyName A 2023-03-26 11:54:35,571 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.HStore(310): Store=70cba8f2e9bfa903d814db3e7b9776da/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:35,571 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,573 DEBUG [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B 2023-03-26 11:54:35,573 DEBUG [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B 2023-03-26 11:54:35,573 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:35,573 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70cba8f2e9bfa903d814db3e7b9776da columnFamilyName B 2023-03-26 11:54:35,573 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.HStore(310): Store=70cba8f2e9bfa903d814db3e7b9776da/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:35,573 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,574 DEBUG [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C 2023-03-26 11:54:35,574 DEBUG [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C 2023-03-26 11:54:35,575 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:54:35,575 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 70cba8f2e9bfa903d814db3e7b9776da columnFamilyName C 2023-03-26 11:54:35,575 INFO [StoreOpener-70cba8f2e9bfa903d814db3e7b9776da-1] regionserver.HStore(310): Store=70cba8f2e9bfa903d814db3e7b9776da/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:54:35,575 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,576 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,577 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:54:35,578 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:35,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:54:35,580 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 70cba8f2e9bfa903d814db3e7b9776da; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69775109, jitterRate=0.039730146527290344}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:54:35,580 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:35,581 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., pid=30, masterSystemTime=1679831675564 2023-03-26 11:54:35,582 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,582 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:35,582 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=70cba8f2e9bfa903d814db3e7b9776da, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:35,583 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831675582"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831675582"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831675582"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831675582"}]},"ts":"1679831675582"} 2023-03-26 11:54:35,585 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-03-26 11:54:35,585 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure 70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 in 171 msec 2023-03-26 11:54:35,586 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-03-26 11:54:35,586 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, ASSIGN in 327 msec 2023-03-26 11:54:35,587 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:54:35,587 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831675587"}]},"ts":"1679831675587"} 2023-03-26 11:54:35,588 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 11:54:35,590 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:54:35,591 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1610 sec 2023-03-26 11:54:36,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-03-26 11:54:36,536 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-03-26 11:54:36,537 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x262f4570 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,541 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@23c97092, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,542 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,543 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37470, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,544 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:54:36,546 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:45764, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:54:36,547 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x00ca898c to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,551 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c124820, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,551 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6585a70a to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,554 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3157d4df, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,554 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x7480f9da to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,558 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3cf5bdb7, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,558 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x1a24d617 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,562 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6919b2a0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,562 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x104dbf4e to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,566 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2ce5573e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,566 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x087b3095 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,572 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ea1ca03, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,572 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x12115052 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,577 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@433f251d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,578 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x7e65578e to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,582 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66ee0bd0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,582 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6fcbaba8 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,588 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b4658f8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,589 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6f8ff99f to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:54:36,592 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5d2b992d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:54:36,597 DEBUG [hconnection-0x4e8f614c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:36,598 DEBUG [hconnection-0x72d60ba3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,600 DEBUG [hconnection-0x2ea4df52-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,600 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37480, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,600 DEBUG [hconnection-0x7ad2e23e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,601 DEBUG [hconnection-0x3f6d3e57-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,601 DEBUG [hconnection-0x56a1795b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,601 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37498, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,602 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37516, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,605 DEBUG [hconnection-0x45dc9b18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,606 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37496, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,609 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37508, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,616 DEBUG [hconnection-0xebdadfe-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,624 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37502, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:36,625 DEBUG [hconnection-0x2de25b08-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,626 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:36,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:36,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:36,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:36,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:36,626 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:36,626 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:36,635 DEBUG [hconnection-0x623945af-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:54:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:36,636 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:36,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:36,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:36,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:36,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,637 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37544, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,638 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:36,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:36,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:36,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,643 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37532, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,643 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37552, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,644 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,644 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:36,646 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:37538, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:54:36,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:36,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:36,649 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:36,649 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:36,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:36,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:36,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:36,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:36,657 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:36,657 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:36,657 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:36,658 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:36,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:36,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:36,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:36,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:36,661 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,661 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,661 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:36,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@753aadb6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:36,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:36,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,666 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:36,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:36,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:36,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:36,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/244ebe9a07d941708af41432027db5b2 2023-03-26 11:54:36,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:36,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:36,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:36,672 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:36,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:36,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:36,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:36,673 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:36,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679831676661, max timestamp=1679831676672Num uniques 859; 2023-03-26 11:54:36,674 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:36,674 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831676661, max timestamp=1679831676672Num uniques 859; 2023-03-26 11:54:36,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1679831676661, max timestamp=1679831676672Num uniques 859; 2023-03-26 11:54:36,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:36,675 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:36,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:36,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:36,679 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:36,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:36,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,680 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:36,680 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:36,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:36,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:36,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:36,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831736687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831736690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831736691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831736692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831736692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,697 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/00677482611a48d481048c4dc0e5ad20 2023-03-26 11:54:36,780 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:36,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:36,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:36,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:36,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:36,783 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:36,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:36,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:36,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:36,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,784 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:36,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:36,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:36,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:36,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:36,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:36,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:36,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:36,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:36,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:36,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:36,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:36,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:36,786 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:36,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:36,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:36,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:36,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:36,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:36,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a062930[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:36,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:36,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,792 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,792 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:36,792 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:36,792 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:36,792 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,792 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:36,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:36,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831736793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:36,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:36,796 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:36,796 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:36,796 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:36,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:36,797 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:36,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831736794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831736795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831736795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,798 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:36,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831736796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:36,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:36,802 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:36,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:36,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:36,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:36,806 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:36,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:36,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,807 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,807 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,807 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,807 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,807 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,807 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:36,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:36,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:36,807 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:36,874 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-26 11:54:36,907 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:36,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:36,910 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:36,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:36,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:36,911 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:36,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:36,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:36,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:36,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:36,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:36,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:36,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:36,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:36,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:36,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:36,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:36,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:36,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:36,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:36,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:36,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:36,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:36,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:36,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:36,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:36,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:36,917 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:36,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37728807[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:36,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:36,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:36,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:36,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:36,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,922 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:36,923 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:36,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:36,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:36,925 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:36,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:36,926 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:36,926 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:36,926 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:36,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:36,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:36,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:36,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:36,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:36,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:36,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:36,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,931 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:36,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:36,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:36,932 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:36,932 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:36,932 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:36,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:36,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831736998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,001 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831736999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831737000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,001 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831737000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,002 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831737000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,032 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:37,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:37,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:37,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:37,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:37,035 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:37,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:37,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:37,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:37,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:37,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:37,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:37,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,039 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:37,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:37,040 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:37,040 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:37,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:37,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:37,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:37,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:37,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:37,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:37,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,042 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:37,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:37,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:37,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5afe1dcf[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:37,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,046 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,046 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:37,046 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:37,046 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:37,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:37,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:37,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:37,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:37,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:37,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:37,050 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:37,050 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:37,050 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:37,050 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:37,050 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:37,053 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,053 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,053 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:37,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,057 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:37,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:37,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:37,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:37,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:37,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:37,059 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:37,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,060 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,060 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,060 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:37,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:37,060 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:37,060 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:37,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=11 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/f752f71d084548998dde7aed432cd5e6 2023-03-26 11:54:37,134 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/244ebe9a07d941708af41432027db5b2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/244ebe9a07d941708af41432027db5b2 2023-03-26 11:54:37,138 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/244ebe9a07d941708af41432027db5b2, entries=150, sequenceid=11, filesize=11.6 K 2023-03-26 11:54:37,138 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/00677482611a48d481048c4dc0e5ad20 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/00677482611a48d481048c4dc0e5ad20 2023-03-26 11:54:37,142 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/00677482611a48d481048c4dc0e5ad20, entries=100, sequenceid=11, filesize=9.3 K 2023-03-26 11:54:37,143 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/f752f71d084548998dde7aed432cd5e6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f752f71d084548998dde7aed432cd5e6 2023-03-26 11:54:37,146 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f752f71d084548998dde7aed432cd5e6, entries=100, sequenceid=11, filesize=9.3 K 2023-03-26 11:54:37,147 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=140.89 KB/144270 for 70cba8f2e9bfa903d814db3e7b9776da in 521ms, sequenceid=11, compaction requested=false 2023-03-26 11:54:37,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:37,160 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:37,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:37,162 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:37,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:37,163 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:37,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:37,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:37,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:37,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:37,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:37,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:37,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:37,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:37,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:37,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:37,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:37,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:37,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:37,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:37,174 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:37,174 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:37,174 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=140.89 KB heapSize=400.22 KB 2023-03-26 11:54:37,174 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:37,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:37,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:37,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:37,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:37,175 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:37,188 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/3cc26d0b88494e9a853f471b8e3da025 2023-03-26 11:54:37,218 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/177686278de5491c9e0f400ad445d2f8 2023-03-26 11:54:37,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:37,303 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:37,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831737305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831737305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,308 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831737306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831737307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,310 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831737307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831737408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831737409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831737409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831737410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,412 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831737410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831737612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831737612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831737613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,614 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831737613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,615 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831737614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,638 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=48 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/f6bc5c8580e74a2ab1afc56ace80b1d7 2023-03-26 11:54:37,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/3cc26d0b88494e9a853f471b8e3da025 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3cc26d0b88494e9a853f471b8e3da025 2023-03-26 11:54:37,648 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3cc26d0b88494e9a853f471b8e3da025, entries=150, sequenceid=48, filesize=11.6 K 2023-03-26 11:54:37,654 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/177686278de5491c9e0f400ad445d2f8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/177686278de5491c9e0f400ad445d2f8 2023-03-26 11:54:37,659 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/177686278de5491c9e0f400ad445d2f8, entries=150, sequenceid=48, filesize=11.6 K 2023-03-26 11:54:37,660 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/f6bc5c8580e74a2ab1afc56ace80b1d7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f6bc5c8580e74a2ab1afc56ace80b1d7 2023-03-26 11:54:37,666 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f6bc5c8580e74a2ab1afc56ace80b1d7, entries=150, sequenceid=48, filesize=11.6 K 2023-03-26 11:54:37,666 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~140.89 KB/144270, heapSize ~400.17 KB/409776, currentSize=40.25 KB/41220 for 70cba8f2e9bfa903d814db3e7b9776da in 492ms, sequenceid=48, compaction requested=false 2023-03-26 11:54:37,667 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:37,667 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:37,667 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:37,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,669 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,670 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:37,672 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a095c3b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:37,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:37,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,673 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:37,674 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:37,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:37,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:37,675 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:37,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:37,679 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:37,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:37,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:37,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:37,679 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:37,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:37,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,683 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,699 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:37,704 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:37,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:37,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:37,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:37,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:37,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:37,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,705 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:37,705 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:37,805 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:37,806 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:37,807 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:37,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:37,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:37,810 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:37,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:37,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:37,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:37,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,812 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:37,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:37,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:37,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:37,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,813 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:37,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:37,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:37,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:37,814 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:37,814 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:37,814 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=40.25 KB heapSize=120.28 KB 2023-03-26 11:54:37,814 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:37,815 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:37,815 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:37,815 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:37,815 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:37,815 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:37,822 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/33c268c4c78144e3ac26ae18f66e08cc 2023-03-26 11:54:37,838 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/a51951ea73324db6bf913bda98f97093 2023-03-26 11:54:37,879 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=13.42 KB at sequenceid=57 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/0c2aa69a511542ea9d47c97dc4326b4f 2023-03-26 11:54:37,884 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/33c268c4c78144e3ac26ae18f66e08cc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/33c268c4c78144e3ac26ae18f66e08cc 2023-03-26 11:54:37,888 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/33c268c4c78144e3ac26ae18f66e08cc, entries=100, sequenceid=57, filesize=9.3 K 2023-03-26 11:54:37,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/a51951ea73324db6bf913bda98f97093 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a51951ea73324db6bf913bda98f97093 2023-03-26 11:54:37,898 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a51951ea73324db6bf913bda98f97093, entries=100, sequenceid=57, filesize=9.3 K 2023-03-26 11:54:37,900 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/0c2aa69a511542ea9d47c97dc4326b4f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/0c2aa69a511542ea9d47c97dc4326b4f 2023-03-26 11:54:37,906 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/0c2aa69a511542ea9d47c97dc4326b4f, entries=100, sequenceid=57, filesize=9.3 K 2023-03-26 11:54:37,907 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~40.25 KB/41220, heapSize ~120.23 KB/123120, currentSize=0 B/0 for 70cba8f2e9bfa903d814db3e7b9776da in 93ms, sequenceid=57, compaction requested=true 2023-03-26 11:54:37,907 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:37,907 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:37,907 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:37,909 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:37,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:37,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@417da5e3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:37,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:37,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:37,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:37,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:37,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:37,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:37,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:37,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:37,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:37,924 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:37,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:37,925 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:37,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:37,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:37,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:37,925 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:37,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:37,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:37,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:37,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:37,926 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:37,926 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:37,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:37,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:37,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:37,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:37,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,930 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:37,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:37,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:37,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:37,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:37,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:37,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:37,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:37,933 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:37,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:37,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:37,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:37,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:37,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:37,934 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:37,934 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:37,940 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/01a88710a9fc461c89d7668b51698433 2023-03-26 11:54:37,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831737944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831737944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,949 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831737945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,950 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831737945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,950 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:37,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831737947, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:37,959 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/9e559c7a73b6492cabca62f1dde23ab0 2023-03-26 11:54:38,034 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:38,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:38,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:38,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:38,040 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:38,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:38,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:38,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,043 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,043 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,043 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:38,044 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:38,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,044 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:38,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:38,049 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:38,049 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,049 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:38,049 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:38,049 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:38,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:38,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,052 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,053 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,053 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831738050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831738050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831738050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:38,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@626c3f7b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:38,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:38,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831738050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,055 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831738051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:38,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:38,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:38,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:38,058 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:38,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:38,061 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:38,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:38,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:38,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:38,061 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:38,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:38,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:38,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,064 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,065 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,070 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,070 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:38,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,071 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:38,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:38,071 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,071 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:38,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:38,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:38,071 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,072 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,072 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,072 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,072 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,072 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,072 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:38,072 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:38,172 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:38,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:38,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:38,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:38,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:38,178 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:38,178 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,178 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:38,178 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:38,179 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,179 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,180 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,180 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,180 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,180 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,181 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:38,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,181 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:38,181 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:38,181 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:38,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:38,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,181 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:38,182 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,182 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:38,182 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:38,182 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:38,182 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:38,183 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,184 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,184 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:38,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d6569c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:38,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:38,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:38,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:38,187 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:38,187 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:38,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:38,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:38,192 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:38,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:38,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:38,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:38,192 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:38,193 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,193 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,193 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:38,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:38,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:38,200 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:38,200 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:38,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:38,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:38,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831738254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831738255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831738255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831738255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831738255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,300 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:38,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:38,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:38,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:38,304 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:38,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:38,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:38,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,308 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,308 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:38,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,309 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:38,309 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:38,309 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:38,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:38,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:38,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:38,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:38,312 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,312 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:38,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:38,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@788e3339[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:38,314 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:38,314 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:38,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:38,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:38,317 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:38,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:38,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:38,320 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:38,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:38,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:38,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:38,320 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:38,323 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,323 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:38,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,323 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:38,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:38,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,325 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:38,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,328 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:38,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:38,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:38,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:38,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:38,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,328 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:38,328 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:38,375 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/b59c3cc9efd94df6905233e0fa14cbcf 2023-03-26 11:54:38,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/01a88710a9fc461c89d7668b51698433 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/01a88710a9fc461c89d7668b51698433 2023-03-26 11:54:38,384 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/01a88710a9fc461c89d7668b51698433, entries=150, sequenceid=70, filesize=11.6 K 2023-03-26 11:54:38,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/9e559c7a73b6492cabca62f1dde23ab0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e559c7a73b6492cabca62f1dde23ab0 2023-03-26 11:54:38,389 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e559c7a73b6492cabca62f1dde23ab0, entries=150, sequenceid=70, filesize=11.6 K 2023-03-26 11:54:38,389 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/b59c3cc9efd94df6905233e0fa14cbcf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b59c3cc9efd94df6905233e0fa14cbcf 2023-03-26 11:54:38,393 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b59c3cc9efd94df6905233e0fa14cbcf, entries=150, sequenceid=70, filesize=11.6 K 2023-03-26 11:54:38,394 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 70cba8f2e9bfa903d814db3e7b9776da in 470ms, sequenceid=70, compaction requested=true 2023-03-26 11:54:38,394 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:38,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:38,394 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:38,394 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:38,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:38,394 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:38,396 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 45196 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:38,396 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:38,396 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:38,396 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:38,396 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,396 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,396 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/244ebe9a07d941708af41432027db5b2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3cc26d0b88494e9a853f471b8e3da025, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/33c268c4c78144e3ac26ae18f66e08cc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/01a88710a9fc461c89d7668b51698433] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=44.1 K 2023-03-26 11:54:38,396 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/00677482611a48d481048c4dc0e5ad20, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/177686278de5491c9e0f400ad445d2f8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a51951ea73324db6bf913bda98f97093, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e559c7a73b6492cabca62f1dde23ab0] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=41.9 K 2023-03-26 11:54:38,397 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 244ebe9a07d941708af41432027db5b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679831676621 2023-03-26 11:54:38,397 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 00677482611a48d481048c4dc0e5ad20, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679831676622 2023-03-26 11:54:38,397 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3cc26d0b88494e9a853f471b8e3da025, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679831676685 2023-03-26 11:54:38,398 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 177686278de5491c9e0f400ad445d2f8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679831676685 2023-03-26 11:54:38,398 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 33c268c4c78144e3ac26ae18f66e08cc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679831677304 2023-03-26 11:54:38,398 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a51951ea73324db6bf913bda98f97093, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679831677304 2023-03-26 11:54:38,399 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 01a88710a9fc461c89d7668b51698433, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679831677922 2023-03-26 11:54:38,399 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9e559c7a73b6492cabca62f1dde23ab0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679831677922 2023-03-26 11:54:38,409 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#218 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:38,409 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:38,423 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/0949f6fe7aaf4698847749df67b99b9b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0949f6fe7aaf4698847749df67b99b9b 2023-03-26 11:54:38,427 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 0949f6fe7aaf4698847749df67b99b9b(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:38,427 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:38,427 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=12, startTime=1679831678394; duration=0sec 2023-03-26 11:54:38,427 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:38,427 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:38,429 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:38,429 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 42856 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:38,429 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:38,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f752f71d084548998dde7aed432cd5e6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f6bc5c8580e74a2ab1afc56ace80b1d7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/0c2aa69a511542ea9d47c97dc4326b4f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b59c3cc9efd94df6905233e0fa14cbcf] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=41.9 K 2023-03-26 11:54:38,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:38,429 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f752f71d084548998dde7aed432cd5e6, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=11, earliestPutTs=1679831676622 2023-03-26 11:54:38,430 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f6bc5c8580e74a2ab1afc56ace80b1d7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=48, earliestPutTs=1679831676685 2023-03-26 11:54:38,431 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0c2aa69a511542ea9d47c97dc4326b4f, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=57, earliestPutTs=1679831677304 2023-03-26 11:54:38,431 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:38,431 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b59c3cc9efd94df6905233e0fa14cbcf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679831677922 2023-03-26 11:54:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:38,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:38,432 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:38,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:38,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:38,433 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,433 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,434 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,434 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:38,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:38,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:38,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:38,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:38,436 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:38,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:38,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:38,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:38,437 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:38,437 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:38,437 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:38,437 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:38,437 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:38,437 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:38,451 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:38,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:38,559 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:38,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831738563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,567 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831738564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,569 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831738566, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831738567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,570 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831738567, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831738668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831738668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,671 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831738670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831738671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,673 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831738671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,819 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/05d454f8ec9d4ab9866a06325a87b748 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/05d454f8ec9d4ab9866a06325a87b748 2023-03-26 11:54:38,823 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into 05d454f8ec9d4ab9866a06325a87b748(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:38,823 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:38,823 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=12, startTime=1679831678394; duration=0sec 2023-03-26 11:54:38,823 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:38,863 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/03fedf569c8a4bfe9268afb4c0e6fdf2 2023-03-26 11:54:38,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831738870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,873 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831738871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831738872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831738873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,877 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:38,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831738874, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:38,888 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/886a2f8e682c4455ac4371bc3e4db001 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/886a2f8e682c4455ac4371bc3e4db001 2023-03-26 11:54:38,894 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 886a2f8e682c4455ac4371bc3e4db001(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:38,894 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:38,895 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=12, startTime=1679831678394; duration=0sec 2023-03-26 11:54:38,895 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:39,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831739174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831739175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831739175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831739179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,181 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831739180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,300 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/9dc7ec1342eb43ce9ad71801e70fbec5 2023-03-26 11:54:39,335 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=90 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/f3a9a9ad3bc940cba29222b85dcd2fc6 2023-03-26 11:54:39,339 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/03fedf569c8a4bfe9268afb4c0e6fdf2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/03fedf569c8a4bfe9268afb4c0e6fdf2 2023-03-26 11:54:39,346 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/03fedf569c8a4bfe9268afb4c0e6fdf2, entries=150, sequenceid=90, filesize=11.6 K 2023-03-26 11:54:39,348 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/9dc7ec1342eb43ce9ad71801e70fbec5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9dc7ec1342eb43ce9ad71801e70fbec5 2023-03-26 11:54:39,353 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9dc7ec1342eb43ce9ad71801e70fbec5, entries=150, sequenceid=90, filesize=11.6 K 2023-03-26 11:54:39,354 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/f3a9a9ad3bc940cba29222b85dcd2fc6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f3a9a9ad3bc940cba29222b85dcd2fc6 2023-03-26 11:54:39,370 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f3a9a9ad3bc940cba29222b85dcd2fc6, entries=150, sequenceid=90, filesize=11.6 K 2023-03-26 11:54:39,371 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 70cba8f2e9bfa903d814db3e7b9776da in 934ms, sequenceid=90, compaction requested=false 2023-03-26 11:54:39,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:39,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:39,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:39,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:39,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:39,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:39,373 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:39,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:39,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:39,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:39,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54f0c376[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:39,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:39,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:39,379 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,379 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:39,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:39,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:39,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:39,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:39,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:39,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:39,386 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:39,386 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:39,386 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:39,386 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:39,386 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:39,391 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,391 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,391 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:39,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:39,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:39,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:39,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:39,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:39,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,397 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:39,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:39,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:39,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:39,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:39,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:39,397 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:39,397 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:39,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:39,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:39,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:39,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:39,498 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:39,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:39,500 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:39,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:39,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:39,502 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:39,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:39,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:39,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:39,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:39,504 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:39,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:39,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:39,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:39,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:39,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:39,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:39,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:39,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:39,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:39,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:39,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:39,508 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:39,508 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:39,508 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:39,508 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:39,508 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:54:39,508 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:39,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:39,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:39,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:39,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:39,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:39,645 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 11:54:39,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:39,679 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:39,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831739691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831739691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831739692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831739692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831739693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,797 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831739796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831739796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831739796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831739796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831739796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,922 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/448cd6ae8ee644c3965153976ab9a4e7 2023-03-26 11:54:39,943 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/ccb70db1c56f491d8f2b2ec38b1f433b 2023-03-26 11:54:39,958 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/cc6837e9aef841829f93d230dab71f15 2023-03-26 11:54:39,963 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/448cd6ae8ee644c3965153976ab9a4e7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/448cd6ae8ee644c3965153976ab9a4e7 2023-03-26 11:54:39,967 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/448cd6ae8ee644c3965153976ab9a4e7, entries=150, sequenceid=105, filesize=11.6 K 2023-03-26 11:54:39,971 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/ccb70db1c56f491d8f2b2ec38b1f433b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/ccb70db1c56f491d8f2b2ec38b1f433b 2023-03-26 11:54:39,977 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/ccb70db1c56f491d8f2b2ec38b1f433b, entries=150, sequenceid=105, filesize=11.6 K 2023-03-26 11:54:39,978 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/cc6837e9aef841829f93d230dab71f15 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/cc6837e9aef841829f93d230dab71f15 2023-03-26 11:54:39,982 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/cc6837e9aef841829f93d230dab71f15, entries=150, sequenceid=105, filesize=11.6 K 2023-03-26 11:54:39,983 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 70cba8f2e9bfa903d814db3e7b9776da in 475ms, sequenceid=105, compaction requested=true 2023-03-26 11:54:39,983 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:39,983 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:39,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:39,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:39,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:39,983 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:39,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:39,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:39,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:39,986 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,986 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:39,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:39,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:39,988 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57e4a974[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:39,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:39,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:39,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:39,989 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:39,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:39,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:39,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:39,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:39,994 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:39,994 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:39,994 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:39,994 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:39,994 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:39,996 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,996 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:39,996 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:39,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:39,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:39,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:39,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:39,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:39,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:40,002 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:40,002 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=114.05 KB heapSize=339.89 KB 2023-03-26 11:54:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831679690, max timestamp=1679831680001Num uniques 859; 2023-03-26 11:54:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:40,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831679690, max timestamp=1679831680001Num uniques 859; 2023-03-26 11:54:40,003 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:40,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:40,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:40,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:40,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:40,004 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:40,004 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:40,015 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/0b7f7a3db02142569862116c43f643e3 2023-03-26 11:54:40,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831740021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,025 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831740021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,026 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831740021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,028 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831740024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,029 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831740026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,102 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,105 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,106 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,106 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:40,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,113 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,113 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,113 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:40,113 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:40,113 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:40,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,118 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:40,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:40,120 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d7c77a2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:40,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:40,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:40,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:40,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:40,121 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,121 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,122 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:40,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:40,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:40,124 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:40,125 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:40,125 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:40,125 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:40,125 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:40,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831740126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831740126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831740126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,131 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,131 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:40,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831740129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831740130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,132 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,133 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,233 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,235 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,236 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,236 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,236 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,236 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,237 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,237 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,238 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,239 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,239 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,239 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:40,239 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,239 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,240 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,240 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,240 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:40,240 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:40,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:40,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,242 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:40,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:40,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24e28ecf[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:40,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:40,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,244 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:40,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:40,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:40,247 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,247 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:40,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:40,250 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:40,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:40,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:40,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:40,250 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:40,251 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,251 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,252 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,253 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,255 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,272 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,272 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,272 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,278 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,278 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:40,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,333 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831740331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831740331, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831740332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831740333, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,336 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831740334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,378 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,381 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,382 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,384 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,385 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,386 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,386 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,387 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,388 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,388 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,388 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:40,388 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:40,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:40,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:40,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@765b1256[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:40,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:40,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:40,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:40,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:40,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:40,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:40,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:40,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:40,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:40,401 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:40,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:40,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:40,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:40,402 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:40,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:40,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,410 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,410 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,441 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/50e3df4f9bdc41bcbc43855d30e297de 2023-03-26 11:54:40,510 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,514 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,517 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,517 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,517 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,518 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:40,518 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,518 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:40,519 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:40,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:40,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,521 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,521 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:40,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@486bae0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:40,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:40,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:40,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:40,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:40,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:40,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:40,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:40,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:40,534 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:40,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:40,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:40,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:40,534 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:40,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,548 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,548 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,548 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831740635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,637 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831740635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831740636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,638 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831740636, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:40,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831740639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,649 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,652 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,652 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,654 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,655 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,655 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:40,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,656 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,657 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,657 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:40,657 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:40,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:40,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:40,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:40,659 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:40,661 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:40,661 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a091cb5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:40,662 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,662 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,662 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:40,662 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:40,662 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:40,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:40,664 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,664 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,664 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,665 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:40,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:40,667 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:40,667 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:40,667 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:40,667 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:40,667 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:40,670 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,671 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,671 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,671 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,676 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,676 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,676 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,676 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,676 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:40,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,676 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,677 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,677 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,777 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,780 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,782 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,783 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,783 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:40,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,783 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,783 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,784 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,784 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,784 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:40,784 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:40,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:40,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:40,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7aad77cd[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:40,788 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:40,788 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,791 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:40,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:40,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:40,792 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,792 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:40,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:40,795 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:40,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:40,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:40,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:40,795 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:40,797 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,797 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,797 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:40,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:40,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:40,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:40,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:40,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:40,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:40,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,801 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:40,801 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:40,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:40,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:40,862 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=126 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/245132a9e01540cb835957613b854f8a 2023-03-26 11:54:40,866 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/0b7f7a3db02142569862116c43f643e3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0b7f7a3db02142569862116c43f643e3 2023-03-26 11:54:40,870 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0b7f7a3db02142569862116c43f643e3, entries=150, sequenceid=126, filesize=11.6 K 2023-03-26 11:54:40,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/50e3df4f9bdc41bcbc43855d30e297de as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/50e3df4f9bdc41bcbc43855d30e297de 2023-03-26 11:54:40,874 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/50e3df4f9bdc41bcbc43855d30e297de, entries=150, sequenceid=126, filesize=11.6 K 2023-03-26 11:54:40,875 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/245132a9e01540cb835957613b854f8a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/245132a9e01540cb835957613b854f8a 2023-03-26 11:54:40,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/245132a9e01540cb835957613b854f8a, entries=150, sequenceid=126, filesize=11.6 K 2023-03-26 11:54:40,880 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~62.62 KB/64120, heapSize ~174.28 KB/178464, currentSize=114.05 KB/116790 for 70cba8f2e9bfa903d814db3e7b9776da in 878ms, sequenceid=126, compaction requested=true 2023-03-26 11:54:40,880 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:40,880 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:40,880 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:40,880 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:40,880 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:40,880 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:40,881 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:40,881 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:40,881 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:40,882 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,882 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:40,882 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/05d454f8ec9d4ab9866a06325a87b748, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9dc7ec1342eb43ce9ad71801e70fbec5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/ccb70db1c56f491d8f2b2ec38b1f433b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/50e3df4f9bdc41bcbc43855d30e297de] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=46.6 K 2023-03-26 11:54:40,882 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,882 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0949f6fe7aaf4698847749df67b99b9b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/03fedf569c8a4bfe9268afb4c0e6fdf2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/448cd6ae8ee644c3965153976ab9a4e7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0b7f7a3db02142569862116c43f643e3] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=46.6 K 2023-03-26 11:54:40,883 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 05d454f8ec9d4ab9866a06325a87b748, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679831677922 2023-03-26 11:54:40,883 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0949f6fe7aaf4698847749df67b99b9b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679831677922 2023-03-26 11:54:40,884 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9dc7ec1342eb43ce9ad71801e70fbec5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1679831677938 2023-03-26 11:54:40,890 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ccb70db1c56f491d8f2b2ec38b1f433b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679831678565 2023-03-26 11:54:40,890 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 03fedf569c8a4bfe9268afb4c0e6fdf2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1679831677938 2023-03-26 11:54:40,890 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 50e3df4f9bdc41bcbc43855d30e297de, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679831679691 2023-03-26 11:54:40,891 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 448cd6ae8ee644c3965153976ab9a4e7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679831678565 2023-03-26 11:54:40,891 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0b7f7a3db02142569862116c43f643e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679831679691 2023-03-26 11:54:40,901 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:40,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:40,903 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:40,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:40,916 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:40,916 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,916 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:40,916 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:40,917 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,917 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,919 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:40,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:40,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:40,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:40,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:40,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:40,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:40,921 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:40,930 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#230 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:40,932 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:40,932 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:40,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:40,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:40,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:40,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:40,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:40,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:40,933 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:54:40,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:40,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:40,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:40,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:40,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:40,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:40,936 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#231 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:40,962 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/f453c9a7905b4e3887465034587ff01e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f453c9a7905b4e3887465034587ff01e 2023-03-26 11:54:40,968 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into f453c9a7905b4e3887465034587ff01e(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:40,968 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:40,968 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=12, startTime=1679831680880; duration=0sec 2023-03-26 11:54:40,969 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:40,969 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:40,971 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:40,971 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:40,971 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:40,971 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/886a2f8e682c4455ac4371bc3e4db001, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f3a9a9ad3bc940cba29222b85dcd2fc6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/cc6837e9aef841829f93d230dab71f15, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/245132a9e01540cb835957613b854f8a] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=46.6 K 2023-03-26 11:54:40,971 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 886a2f8e682c4455ac4371bc3e4db001, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1679831677922 2023-03-26 11:54:40,972 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f3a9a9ad3bc940cba29222b85dcd2fc6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=90, earliestPutTs=1679831677938 2023-03-26 11:54:40,972 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting cc6837e9aef841829f93d230dab71f15, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679831678565 2023-03-26 11:54:40,973 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 245132a9e01540cb835957613b854f8a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679831679691 2023-03-26 11:54:40,977 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/1ef21589146b499cb9992b215b90ef02 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1ef21589146b499cb9992b215b90ef02 2023-03-26 11:54:40,985 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 1ef21589146b499cb9992b215b90ef02(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:40,985 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:40,985 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=12, startTime=1679831680880; duration=0sec 2023-03-26 11:54:40,985 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:40,992 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#233 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:41,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:41,141 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:41,150 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831741146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,150 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831741147, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831741148, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831741150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831741150, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,253 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831741252, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831741255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831741255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831741255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,256 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,257 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831741255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,390 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/68085721b6a145caa3e97e589566cfd6 2023-03-26 11:54:41,405 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/d545b04a05694d76b0e9650273a6aabf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d545b04a05694d76b0e9650273a6aabf 2023-03-26 11:54:41,409 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into d545b04a05694d76b0e9650273a6aabf(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:41,409 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:41,410 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=12, startTime=1679831680880; duration=0sec 2023-03-26 11:54:41,410 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831741456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,457 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831741457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831741458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831741458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,460 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831741458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,760 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831741759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831741759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831741760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831741760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:41,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831741761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,800 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/92dfa2332e464574aef9eaf148b9aa37 2023-03-26 11:54:41,811 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/ba640aaf185e4e98ac96786ebf166604 2023-03-26 11:54:41,814 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/68085721b6a145caa3e97e589566cfd6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/68085721b6a145caa3e97e589566cfd6 2023-03-26 11:54:41,818 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/68085721b6a145caa3e97e589566cfd6, entries=150, sequenceid=146, filesize=11.8 K 2023-03-26 11:54:41,819 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/92dfa2332e464574aef9eaf148b9aa37 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/92dfa2332e464574aef9eaf148b9aa37 2023-03-26 11:54:41,824 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/92dfa2332e464574aef9eaf148b9aa37, entries=150, sequenceid=146, filesize=11.8 K 2023-03-26 11:54:41,825 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/ba640aaf185e4e98ac96786ebf166604 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ba640aaf185e4e98ac96786ebf166604 2023-03-26 11:54:41,833 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ba640aaf185e4e98ac96786ebf166604, entries=150, sequenceid=146, filesize=11.8 K 2023-03-26 11:54:41,833 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.StoreScanner(961): StoreScanner already closing. There is no need to updateReaders 2023-03-26 11:54:41,834 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 70cba8f2e9bfa903d814db3e7b9776da in 901ms, sequenceid=146, compaction requested=false 2023-03-26 11:54:41,834 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:41,834 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:41,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:41,836 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:41,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:41,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:41,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:41,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54ca136c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:41,838 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:41,838 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,845 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:41,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:41,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:41,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:41,849 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,849 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:41,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:41,849 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:41,849 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:41,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:41,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:41,852 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:41,853 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:41,853 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:41,853 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:41,853 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:41,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:41,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:41,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:41,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:41,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:41,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:41,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:41,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:41,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:41,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:41,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:41,860 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:41,861 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:41,861 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:41,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:41,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:41,961 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:41,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:41,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:41,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:41,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:41,964 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:41,964 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:41,964 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:41,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:41,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:41,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:41,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:41,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:41,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:41,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:41,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:41,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:41,967 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:41,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:41,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:41,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:41,968 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:41,968 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:41,968 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:54:41,968 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:41,968 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:41,968 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:41,969 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:41,969 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:41,969 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:42,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:42,264 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:42,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831742274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831742276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831742277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831742277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831742278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,376 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/4484a252be9f4aa880f30f6f091790a5 2023-03-26 11:54:42,380 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831742378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831742379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831742381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831742381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831742381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831742582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831742583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,585 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831742583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831742585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,589 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831742586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,787 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/3f24ecd1b8d94e6ba5dac7b8304e584a 2023-03-26 11:54:42,797 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=162 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/6f264318a1384e8691479cba975935b4 2023-03-26 11:54:42,801 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/4484a252be9f4aa880f30f6f091790a5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/4484a252be9f4aa880f30f6f091790a5 2023-03-26 11:54:42,805 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/4484a252be9f4aa880f30f6f091790a5, entries=150, sequenceid=162, filesize=11.8 K 2023-03-26 11:54:42,806 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/3f24ecd1b8d94e6ba5dac7b8304e584a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/3f24ecd1b8d94e6ba5dac7b8304e584a 2023-03-26 11:54:42,811 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/3f24ecd1b8d94e6ba5dac7b8304e584a, entries=150, sequenceid=162, filesize=11.8 K 2023-03-26 11:54:42,811 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/6f264318a1384e8691479cba975935b4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f264318a1384e8691479cba975935b4 2023-03-26 11:54:42,815 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f264318a1384e8691479cba975935b4, entries=150, sequenceid=162, filesize=11.8 K 2023-03-26 11:54:42,816 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 70cba8f2e9bfa903d814db3e7b9776da in 848ms, sequenceid=162, compaction requested=true 2023-03-26 11:54:42,816 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:42,816 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:42,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,818 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,818 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:42,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:42,819 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65aae2af[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:42,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:42,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,821 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:42,821 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:42,821 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,821 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:42,822 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:42,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:42,822 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:42,824 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:42,824 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:42,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:42,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:42,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:42,825 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:42,827 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,827 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,827 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:42,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:42,828 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:42,829 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:42,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:42,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:42,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:42,831 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:42,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:42,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:42,832 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:42,832 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:42,832 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:42,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:42,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:42,886 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-26 11:54:42,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:42,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:42,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:42,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:42,887 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:42,887 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:42,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831742895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831742895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831742896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831742896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,899 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:42,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831742896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,932 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:42,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:42,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:42,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:42,935 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:42,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:42,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:42,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:42,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,937 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:42,937 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:42,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:42,938 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,938 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:42,938 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:42,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:42,939 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:42,939 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:42,939 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:42,939 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:42,939 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:42,939 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:42,939 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:42,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:42,941 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:42,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:42,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:42,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6744a61b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:42,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:42,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,946 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:42,946 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:42,946 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:42,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:42,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,947 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:42,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:42,947 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:42,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:42,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:42,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:42,950 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:42,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:42,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:42,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:42,950 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:42,952 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,952 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:42,952 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:42,952 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:42,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:42,953 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:42,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:42,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:42,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:42,956 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:42,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:42,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:42,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:42,956 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,956 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:42,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,957 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:42,957 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831743000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831743000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,003 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831743000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,003 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831743000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,004 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831743000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,056 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,060 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,064 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:43,064 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,065 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,065 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,065 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:43,065 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:43,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:43,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,066 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:43,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7646b621[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:43,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:43,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,069 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:43,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:43,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:43,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:43,071 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:43,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:43,074 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:43,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:43,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:43,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:43,074 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:43,076 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,076 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,076 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,076 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,082 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,082 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,082 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:43,083 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:43,082 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:43,083 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:43,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,083 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:43,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,083 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,183 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,186 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,187 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,187 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,188 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,189 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,190 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:43,190 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,190 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,190 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,190 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:43,190 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:43,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:43,193 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:43,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:43,195 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72938af6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:43,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:43,195 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,196 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:43,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:43,196 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:43,200 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:43,200 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:43,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:43,201 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:43,201 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:43,201 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:43,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,203 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,205 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831743204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831743204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831743205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,207 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831743205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831743205, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:43,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:43,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,210 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:43,210 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:43,294 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/1be584f1b6984fd5ae36bb9fedf8c00e 2023-03-26 11:54:43,310 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,313 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,313 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,313 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,313 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,314 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,314 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,316 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,316 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:43,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,317 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,317 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,317 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:43,317 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:43,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:43,318 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:43,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@409ea191[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:43,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:43,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:43,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,322 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:43,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:43,324 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:43,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:43,326 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:43,326 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:43,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:43,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:43,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:43,326 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:43,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,328 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,329 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:43,332 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:43,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,332 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:43,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,332 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:43,332 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:43,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,432 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,436 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,437 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,437 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:43,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,439 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,439 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,439 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:43,439 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:43,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:43,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:43,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:43,442 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:43,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:43,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@228b3e7[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:43,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:43,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,445 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:43,445 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:43,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:43,447 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:43,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:43,449 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:43,449 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:43,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:43,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:43,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:43,449 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:43,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,453 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,458 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:43,458 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:43,458 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:43,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,459 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:43,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,459 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:43,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831743507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831743508, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831743509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:43,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831743509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831743509, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,559 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,563 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,565 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:43,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:43,567 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:43,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:43,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:43,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,569 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:43,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:43,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@411cfa3d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:43,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:43,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,573 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:43,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:43,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:43,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:43,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,574 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:43,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:43,578 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:43,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:43,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:43,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:43,578 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:43,597 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,597 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,597 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,602 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:43,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,602 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:43,602 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:43,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:43,703 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/2ba6d7f8cbb94bd4a44dace128b69edf 2023-03-26 11:54:43,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,707 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,710 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,712 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,712 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,712 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:43,713 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:43,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:43,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,715 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:43,718 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:43,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f361cd2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:43,718 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/8fce94974b0b47bfbf7f13801a473ed0 2023-03-26 11:54:43,719 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:43,719 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:43,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:43,721 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,721 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:43,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:43,724 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:43,724 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:43,724 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:43,724 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:43,724 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:43,724 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/1be584f1b6984fd5ae36bb9fedf8c00e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1be584f1b6984fd5ae36bb9fedf8c00e 2023-03-26 11:54:43,728 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1be584f1b6984fd5ae36bb9fedf8c00e, entries=150, sequenceid=183, filesize=11.8 K 2023-03-26 11:54:43,728 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/2ba6d7f8cbb94bd4a44dace128b69edf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2ba6d7f8cbb94bd4a44dace128b69edf 2023-03-26 11:54:43,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,731 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,731 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:43,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:43,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,732 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2ba6d7f8cbb94bd4a44dace128b69edf, entries=150, sequenceid=183, filesize=11.8 K 2023-03-26 11:54:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:43,733 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/8fce94974b0b47bfbf7f13801a473ed0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8fce94974b0b47bfbf7f13801a473ed0 2023-03-26 11:54:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,734 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:43,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:43,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,737 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:43,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:43,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,737 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8fce94974b0b47bfbf7f13801a473ed0, entries=150, sequenceid=183, filesize=11.8 K 2023-03-26 11:54:43,737 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:43,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:43,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:43,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:43,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,737 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:43,738 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:43,738 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 70cba8f2e9bfa903d814db3e7b9776da in 852ms, sequenceid=183, compaction requested=true 2023-03-26 11:54:43,738 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:43,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:43,738 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:43,738 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:43,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:43,738 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:43,739 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48268 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:43,740 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:43,740 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,740 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1ef21589146b499cb9992b215b90ef02, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/68085721b6a145caa3e97e589566cfd6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/4484a252be9f4aa880f30f6f091790a5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1be584f1b6984fd5ae36bb9fedf8c00e] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=47.1 K 2023-03-26 11:54:43,740 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48268 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:43,740 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:43,740 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,740 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f453c9a7905b4e3887465034587ff01e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/92dfa2332e464574aef9eaf148b9aa37, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/3f24ecd1b8d94e6ba5dac7b8304e584a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2ba6d7f8cbb94bd4a44dace128b69edf] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=47.1 K 2023-03-26 11:54:43,741 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1ef21589146b499cb9992b215b90ef02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679831679691 2023-03-26 11:54:43,741 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f453c9a7905b4e3887465034587ff01e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679831679691 2023-03-26 11:54:43,741 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 68085721b6a145caa3e97e589566cfd6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679831680019 2023-03-26 11:54:43,741 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 92dfa2332e464574aef9eaf148b9aa37, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679831680019 2023-03-26 11:54:43,741 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 4484a252be9f4aa880f30f6f091790a5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679831681146 2023-03-26 11:54:43,741 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3f24ecd1b8d94e6ba5dac7b8304e584a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679831681146 2023-03-26 11:54:43,742 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1be584f1b6984fd5ae36bb9fedf8c00e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831682268 2023-03-26 11:54:43,742 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2ba6d7f8cbb94bd4a44dace128b69edf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831682268 2023-03-26 11:54:43,750 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#242 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:43,750 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#243 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:43,772 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/db188b8e12304063b0ae03ef91e12f5f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/db188b8e12304063b0ae03ef91e12f5f 2023-03-26 11:54:43,777 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into db188b8e12304063b0ae03ef91e12f5f(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:43,777 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:43,777 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=12, startTime=1679831683738; duration=0sec 2023-03-26 11:54:43,777 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:43,777 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:43,778 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48268 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:43,778 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:43,778 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,778 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d545b04a05694d76b0e9650273a6aabf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ba640aaf185e4e98ac96786ebf166604, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f264318a1384e8691479cba975935b4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8fce94974b0b47bfbf7f13801a473ed0] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=47.1 K 2023-03-26 11:54:43,779 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d545b04a05694d76b0e9650273a6aabf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=126, earliestPutTs=1679831679691 2023-03-26 11:54:43,779 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ba640aaf185e4e98ac96786ebf166604, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1679831680019 2023-03-26 11:54:43,779 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6f264318a1384e8691479cba975935b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=162, earliestPutTs=1679831681146 2023-03-26 11:54:43,780 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8fce94974b0b47bfbf7f13801a473ed0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831682268 2023-03-26 11:54:43,787 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#244 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:43,796 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/da3e350050624988bdbba54ff90d2d7e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/da3e350050624988bdbba54ff90d2d7e 2023-03-26 11:54:43,799 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into da3e350050624988bdbba54ff90d2d7e(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:43,800 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:43,800 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=12, startTime=1679831683738; duration=0sec 2023-03-26 11:54:43,800 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:43,838 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:43,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:43,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:43,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:43,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:43,841 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:43,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:43,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:43,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:43,842 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:43,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:43,843 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:43,843 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:43,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:43,844 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:43,844 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:43,844 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:43,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:43,844 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:43,844 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:43,844 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:43,845 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:43,853 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/f3ebabdf58ba411ba54dfc56162a23bf 2023-03-26 11:54:43,864 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/080ceffcceb94070aa42177a07053a79 2023-03-26 11:54:44,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:44,014 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:44,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831744024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831684022, max timestamp=1679831684026Num uniques 859; 2023-03-26 11:54:44,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:44,028 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831684022, max timestamp=1679831684026Num uniques 859; 2023-03-26 11:54:44,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831684022, max timestamp=1679831684026Num uniques 859; 2023-03-26 11:54:44,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:44,029 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:54:44,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831744039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831744038, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831744042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,045 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831744042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,130 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831744128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831744143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,145 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831744143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831744146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831744146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,166 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/ce2e7de6b7584882b111328963c543ce as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ce2e7de6b7584882b111328963c543ce 2023-03-26 11:54:44,170 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into ce2e7de6b7584882b111328963c543ce(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:44,170 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:44,170 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=12, startTime=1679831683738; duration=0sec 2023-03-26 11:54:44,170 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:44,290 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/8bfcc702fb6e41b48fc29aa5b9cd317b 2023-03-26 11:54:44,295 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/f3ebabdf58ba411ba54dfc56162a23bf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f3ebabdf58ba411ba54dfc56162a23bf 2023-03-26 11:54:44,299 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f3ebabdf58ba411ba54dfc56162a23bf, entries=150, sequenceid=197, filesize=11.8 K 2023-03-26 11:54:44,299 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/080ceffcceb94070aa42177a07053a79 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/080ceffcceb94070aa42177a07053a79 2023-03-26 11:54:44,304 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/080ceffcceb94070aa42177a07053a79, entries=150, sequenceid=197, filesize=11.8 K 2023-03-26 11:54:44,304 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/8bfcc702fb6e41b48fc29aa5b9cd317b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8bfcc702fb6e41b48fc29aa5b9cd317b 2023-03-26 11:54:44,313 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8bfcc702fb6e41b48fc29aa5b9cd317b, entries=150, sequenceid=197, filesize=11.8 K 2023-03-26 11:54:44,314 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 70cba8f2e9bfa903d814db3e7b9776da in 470ms, sequenceid=197, compaction requested=false 2023-03-26 11:54:44,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:44,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:44,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:44,317 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:44,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:44,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:44,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:44,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a91a28c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:44,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:44,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:44,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:44,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:44,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:44,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:44,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:44,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:44,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:44,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:44,334 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:44,334 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-26 11:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:44,335 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:44,335 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:44,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:44,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:44,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:44,335 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:44,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:44,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:44,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,340 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:44,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:44,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:44,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:44,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:44,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:44,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:44,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:44,351 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:44,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:44,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:44,351 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:44,352 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:44,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:44,351 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,352 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:44,352 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,352 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,357 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831744353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,359 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/27a7dcec4272422f9cd655c256055d08 2023-03-26 11:54:44,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831744355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831744360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831744361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831744361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/0816f12a438941abb13964cd81cfa352 2023-03-26 11:54:44,446 WARN [DataStreamer for file /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/a38e648d99c14b33acc45c183395602c] hdfs.DataStreamer(982): Caught exception java.lang.InterruptedException at java.lang.Object.wait(Native Method) at java.lang.Thread.join(Thread.java:1257) at java.lang.Thread.join(Thread.java:1331) at org.apache.hadoop.hdfs.DataStreamer.closeResponder(DataStreamer.java:980) at org.apache.hadoop.hdfs.DataStreamer.endBlock(DataStreamer.java:630) at org.apache.hadoop.hdfs.DataStreamer.run(DataStreamer.java:807) 2023-03-26 11:54:44,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=232 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/a38e648d99c14b33acc45c183395602c 2023-03-26 11:54:44,452 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/27a7dcec4272422f9cd655c256055d08 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/27a7dcec4272422f9cd655c256055d08 2023-03-26 11:54:44,452 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:44,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:44,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:44,458 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/27a7dcec4272422f9cd655c256055d08, entries=200, sequenceid=232, filesize=14.1 K 2023-03-26 11:54:44,459 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/0816f12a438941abb13964cd81cfa352 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/0816f12a438941abb13964cd81cfa352 2023-03-26 11:54:44,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:44,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:44,460 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:44,460 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:44,460 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:44,460 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:44,460 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,460 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,463 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/0816f12a438941abb13964cd81cfa352, entries=150, sequenceid=232, filesize=11.8 K 2023-03-26 11:54:44,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831744458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,465 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/a38e648d99c14b33acc45c183395602c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/a38e648d99c14b33acc45c183395602c 2023-03-26 11:54:44,467 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:44,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:44,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:44,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:44,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:44,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:44,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:44,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:44,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:44,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:44,472 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,472 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:44,472 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:44,472 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:44,472 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:44,472 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,472 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831744462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831744462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831744464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/a38e648d99c14b33acc45c183395602c, entries=150, sequenceid=232, filesize=11.8 K 2023-03-26 11:54:44,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831744462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,474 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:44,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:44,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=60.38 KB/61830 for 70cba8f2e9bfa903d814db3e7b9776da in 141ms, sequenceid=232, compaction requested=true 2023-03-26 11:54:44,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:44,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:44,475 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:44,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:44,475 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:44,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:44,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,476 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:44,477 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38908 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:44,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:44,477 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:44,477 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,477 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,477 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/db188b8e12304063b0ae03ef91e12f5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/080ceffcceb94070aa42177a07053a79, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/0816f12a438941abb13964cd81cfa352] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=35.7 K 2023-03-26 11:54:44,477 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ce2e7de6b7584882b111328963c543ce, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f3ebabdf58ba411ba54dfc56162a23bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/27a7dcec4272422f9cd655c256055d08] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=38.0 K 2023-03-26 11:54:44,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:44,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting db188b8e12304063b0ae03ef91e12f5f, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831682268 2023-03-26 11:54:44,478 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ce2e7de6b7584882b111328963c543ce, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831682268 2023-03-26 11:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:44,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d619db2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:44,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:44,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,478 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 080ceffcceb94070aa42177a07053a79, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831682894 2023-03-26 11:54:44,478 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f3ebabdf58ba411ba54dfc56162a23bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831682894 2023-03-26 11:54:44,479 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 0816f12a438941abb13964cd81cfa352, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679831684038 2023-03-26 11:54:44,479 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 27a7dcec4272422f9cd655c256055d08, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679831684038 2023-03-26 11:54:44,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:44,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:44,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:44,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:44,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:44,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:44,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:44,483 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:44,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:44,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:44,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:44,483 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:44,485 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,485 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,485 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:44,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:44,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,486 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:44,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:44,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:44,490 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#251 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:44,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:44,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:44,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:44,492 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:44,492 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:44,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:44,492 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:44,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:44,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:44,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:44,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:44,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:44,496 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#252 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:44,537 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/341ab11293c34fd99d4505cd036007dc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/341ab11293c34fd99d4505cd036007dc 2023-03-26 11:54:44,539 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/c651f78c0ed44299aff0d1c66fce893e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c651f78c0ed44299aff0d1c66fce893e 2023-03-26 11:54:44,543 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 341ab11293c34fd99d4505cd036007dc(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:44,543 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:44,543 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=13, startTime=1679831684475; duration=0sec 2023-03-26 11:54:44,543 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:44,543 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:44,545 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36518 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:44,545 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:44,545 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,545 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/da3e350050624988bdbba54ff90d2d7e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8bfcc702fb6e41b48fc29aa5b9cd317b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/a38e648d99c14b33acc45c183395602c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=35.7 K 2023-03-26 11:54:44,546 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting da3e350050624988bdbba54ff90d2d7e, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831682268 2023-03-26 11:54:44,546 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8bfcc702fb6e41b48fc29aa5b9cd317b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831682894 2023-03-26 11:54:44,547 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a38e648d99c14b33acc45c183395602c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679831684038 2023-03-26 11:54:44,557 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into c651f78c0ed44299aff0d1c66fce893e(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:44,557 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:44,557 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=13, startTime=1679831684475; duration=0sec 2023-03-26 11:54:44,557 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:44,586 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#253 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:44,592 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:44,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:44,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:44,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:44,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:44,605 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:44,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:44,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:44,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:44,607 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,607 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:44,608 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:44,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:44,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,612 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:44,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:44,613 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:44,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:44,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:44,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:44,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:44,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:44,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:44,628 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:44,628 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:44,628 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:44,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:44,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:44,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:44,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:44,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:44,630 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:44,665 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/8ba2318e579248b6829175e2a2a429e9 2023-03-26 11:54:44,669 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:44,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:44,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831744690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,695 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831744690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,696 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831744691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,696 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831744692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831744691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831744797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,798 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831744797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,801 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831744799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,801 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831744798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:44,802 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:44,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831744799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831745000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831745000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,007 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/c7e56d7733014a73be2665e042794516 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c7e56d7733014a73be2665e042794516 2023-03-26 11:54:45,008 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831745008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831745008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,009 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831745008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,013 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into c7e56d7733014a73be2665e042794516(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:45,013 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:45,013 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=13, startTime=1679831684475; duration=0sec 2023-03-26 11:54:45,013 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:45,122 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/30dd23adccdd4bb78acaaf7a7ab846de 2023-03-26 11:54:45,139 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=246 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/b1823c6ed85e4633aad39004a863b385 2023-03-26 11:54:45,146 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/8ba2318e579248b6829175e2a2a429e9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8ba2318e579248b6829175e2a2a429e9 2023-03-26 11:54:45,150 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8ba2318e579248b6829175e2a2a429e9, entries=150, sequenceid=246, filesize=11.8 K 2023-03-26 11:54:45,150 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/30dd23adccdd4bb78acaaf7a7ab846de as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/30dd23adccdd4bb78acaaf7a7ab846de 2023-03-26 11:54:45,156 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/30dd23adccdd4bb78acaaf7a7ab846de, entries=150, sequenceid=246, filesize=11.8 K 2023-03-26 11:54:45,157 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/b1823c6ed85e4633aad39004a863b385 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b1823c6ed85e4633aad39004a863b385 2023-03-26 11:54:45,161 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b1823c6ed85e4633aad39004a863b385, entries=150, sequenceid=246, filesize=11.8 K 2023-03-26 11:54:45,161 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 70cba8f2e9bfa903d814db3e7b9776da in 533ms, sequenceid=246, compaction requested=false 2023-03-26 11:54:45,162 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:45,162 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:45,162 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:45,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:45,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b8a4931[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:45,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:45,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,167 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:45,167 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:45,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:45,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,169 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:45,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:45,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:45,172 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:45,172 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:45,172 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:45,172 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:45,172 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:45,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,174 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,175 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,179 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:45,179 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,180 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,179 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,180 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:45,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,180 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:45,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,280 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:45,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:45,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:45,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:45,285 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:45,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:45,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:45,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:45,288 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,288 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:45,288 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:45,288 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:45,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:45,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:45,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:45,292 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:54:45,292 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:45,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:45,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:45,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:45,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:45,293 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:45,301 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/f0d8e9c78eb24186b1e43bff02dd193a 2023-03-26 11:54:45,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:45,306 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:45,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831745317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831745317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,320 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37508 deadline: 1679831745317, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37480 deadline: 1679831745318, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37516 deadline: 1679831745319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,327 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/f1e2ee6e13b642b3abd60306db5d8e85 2023-03-26 11:54:45,350 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/084b148255c64138af39d212e008c32d 2023-03-26 11:54:45,354 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/f0d8e9c78eb24186b1e43bff02dd193a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f0d8e9c78eb24186b1e43bff02dd193a 2023-03-26 11:54:45,358 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f0d8e9c78eb24186b1e43bff02dd193a, entries=150, sequenceid=267, filesize=11.9 K 2023-03-26 11:54:45,359 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/f1e2ee6e13b642b3abd60306db5d8e85 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f1e2ee6e13b642b3abd60306db5d8e85 2023-03-26 11:54:45,363 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f1e2ee6e13b642b3abd60306db5d8e85, entries=150, sequenceid=267, filesize=11.9 K 2023-03-26 11:54:45,363 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/084b148255c64138af39d212e008c32d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/084b148255c64138af39d212e008c32d 2023-03-26 11:54:45,372 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/084b148255c64138af39d212e008c32d, entries=150, sequenceid=267, filesize=11.9 K 2023-03-26 11:54:45,373 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 70cba8f2e9bfa903d814db3e7b9776da in 81ms, sequenceid=267, compaction requested=true 2023-03-26 11:54:45,373 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:45,373 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:45,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:45,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:45,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:45,383 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c51a1e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:45,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:45,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,385 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:45,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:45,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:45,387 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,387 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:45,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:45,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:45,390 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:45,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:45,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:45,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:45,390 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:45,392 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,399 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:45,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:45,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,400 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:45,400 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:45,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:45,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:45,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 11:54:45,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:45,425 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:45,425 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:45,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:45,426 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:45,426 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:45,438 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/3dd3498d26ab4555af426634179b7fde 2023-03-26 11:54:45,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831745442, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,446 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831745443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,447 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831745445, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,500 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:45,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:45,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:45,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:45,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:45,504 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:45,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:45,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:45,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:45,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:45,508 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:45,508 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:45,508 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:45,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:45,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:45,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:45,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:45,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,513 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:45,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:45,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73d741c8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:45,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:45,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:45,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:45,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:45,517 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:45,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:45,520 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:45,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:45,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:45,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:45,520 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:45,522 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,522 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,522 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:45,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:45,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,526 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:45,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,526 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:45,526 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:45,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831745546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831745546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831745548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,626 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:45,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:45,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:45,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:45,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:45,630 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:45,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:45,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:45,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,632 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:45,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:45,633 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:45,633 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:45,633 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,633 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:45,633 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:45,633 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:45,633 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:45,634 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:45,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:45,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:45,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@220ace0d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:45,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:45,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,639 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,639 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:45,639 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:45,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:45,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:45,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:45,643 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:45,643 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:45,643 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:45,643 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:45,643 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:45,644 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,644 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,644 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:45,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:45,648 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:45,648 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,648 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:45,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,649 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,748 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:45,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:45,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:45,751 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831745751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831745751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:45,753 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:45,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:45,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:45,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831745751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,755 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,755 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,756 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,756 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:45,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,756 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:45,757 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:45,757 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:45,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:45,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,757 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:45,761 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,761 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:45,761 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:45,761 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:45,761 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:45,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,763 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:45,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@be0c8bc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:45,766 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:45,766 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:45,768 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:45,768 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,768 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,769 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:45,769 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:45,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:45,772 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:45,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:45,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:45,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:45,772 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:45,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,776 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,778 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:45,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,782 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:45,782 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:45,782 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:45,821 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831745820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:45,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831745821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,851 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/be36d797b6bd46c29d856e82c5bc16b7 2023-03-26 11:54:45,882 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:45,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:45,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:45,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:45,885 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:45,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:45,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:45,886 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,886 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,887 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,887 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,887 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,887 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:45,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:45,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:45,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:45,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:45,889 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:45,889 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:45,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,890 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:45,892 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47393634[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:45,892 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:45,892 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:45,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:45,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:45,894 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:45,898 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,898 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:45,898 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:45,901 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:45,901 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:45,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:45,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:45,901 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:45,901 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:45,902 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:45,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:45,902 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,902 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:45,903 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:45,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:45,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:45,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:45,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:45,906 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:45,906 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:45,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:45,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:45,906 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:45,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:45,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:45,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:45,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:45,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:45,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:45,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,007 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,009 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,010 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,013 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,013 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,014 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,014 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,014 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,014 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:46,014 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,014 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,017 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e29f61a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,017 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,019 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,020 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,020 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,021 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,021 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,023 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,023 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,023 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,026 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,026 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,026 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:46,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,030 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,030 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831746052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831746054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,059 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831746056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,130 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,134 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,135 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,136 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,136 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,136 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,137 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,137 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:46,137 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,137 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,137 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,137 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,137 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:46,137 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,138 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a272b74[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,141 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,141 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,143 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,143 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,147 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,147 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,148 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,148 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,148 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,150 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,152 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,152 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,152 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,152 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,152 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,153 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,153 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,153 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:46,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,253 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,255 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,256 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,256 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,256 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,257 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,257 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,257 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,260 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,260 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,261 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,261 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,261 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,261 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,262 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:46,262 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,263 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,264 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,266 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=282 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/dc871f23aae74bf89a5c97ac7ae10c43 2023-03-26 11:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@778fa9a4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,267 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,267 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,271 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,273 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/3dd3498d26ab4555af426634179b7fde as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3dd3498d26ab4555af426634179b7fde 2023-03-26 11:54:46,278 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3dd3498d26ab4555af426634179b7fde, entries=150, sequenceid=282, filesize=11.9 K 2023-03-26 11:54:46,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,279 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/be36d797b6bd46c29d856e82c5bc16b7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/be36d797b6bd46c29d856e82c5bc16b7 2023-03-26 11:54:46,283 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/be36d797b6bd46c29d856e82c5bc16b7, entries=150, sequenceid=282, filesize=11.9 K 2023-03-26 11:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:46,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,285 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,285 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,291 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/dc871f23aae74bf89a5c97ac7ae10c43 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/dc871f23aae74bf89a5c97ac7ae10c43 2023-03-26 11:54:46,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,293 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,295 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,297 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/dc871f23aae74bf89a5c97ac7ae10c43, entries=150, sequenceid=282, filesize=11.9 K 2023-03-26 11:54:46,298 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 70cba8f2e9bfa903d814db3e7b9776da in 874ms, sequenceid=282, compaction requested=true 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,298 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:46,298 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:46,299 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 11:54:46,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,298 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,298 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,298 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,299 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:46,298 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:46,299 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,299 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,300 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:46,300 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:46,300 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,301 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/341ab11293c34fd99d4505cd036007dc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8ba2318e579248b6829175e2a2a429e9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f0d8e9c78eb24186b1e43bff02dd193a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3dd3498d26ab4555af426634179b7fde] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=47.8 K 2023-03-26 11:54:46,301 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:46,301 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:46,301 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,301 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c651f78c0ed44299aff0d1c66fce893e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/30dd23adccdd4bb78acaaf7a7ab846de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f1e2ee6e13b642b3abd60306db5d8e85, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/be36d797b6bd46c29d856e82c5bc16b7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=47.8 K 2023-03-26 11:54:46,301 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 341ab11293c34fd99d4505cd036007dc, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679831684038 2023-03-26 11:54:46,302 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c651f78c0ed44299aff0d1c66fce893e, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679831684038 2023-03-26 11:54:46,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8ba2318e579248b6829175e2a2a429e9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1679831684338 2023-03-26 11:54:46,302 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 30dd23adccdd4bb78acaaf7a7ab846de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1679831684338 2023-03-26 11:54:46,302 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f0d8e9c78eb24186b1e43bff02dd193a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679831684688 2023-03-26 11:54:46,302 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f1e2ee6e13b642b3abd60306db5d8e85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679831684688 2023-03-26 11:54:46,303 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3dd3498d26ab4555af426634179b7fde, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679831685314 2023-03-26 11:54:46,303 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting be36d797b6bd46c29d856e82c5bc16b7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679831685314 2023-03-26 11:54:46,326 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:46,327 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#263 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:46,363 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/61efb34b34b544deb86d0f79e6c04017 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/61efb34b34b544deb86d0f79e6c04017 2023-03-26 11:54:46,368 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/35d2e192ceb54ce8a63a4c46693e60e1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/35d2e192ceb54ce8a63a4c46693e60e1 2023-03-26 11:54:46,371 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 61efb34b34b544deb86d0f79e6c04017(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:46,371 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,371 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=12, startTime=1679831686298; duration=0sec 2023-03-26 11:54:46,371 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:46,371 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:46,372 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48956 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:46,373 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:46,373 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,373 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c7e56d7733014a73be2665e042794516, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b1823c6ed85e4633aad39004a863b385, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/084b148255c64138af39d212e008c32d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/dc871f23aae74bf89a5c97ac7ae10c43] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=47.8 K 2023-03-26 11:54:46,373 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c7e56d7733014a73be2665e042794516, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=232, earliestPutTs=1679831684038 2023-03-26 11:54:46,374 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b1823c6ed85e4633aad39004a863b385, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=246, earliestPutTs=1679831684338 2023-03-26 11:54:46,376 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 084b148255c64138af39d212e008c32d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679831684688 2023-03-26 11:54:46,376 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dc871f23aae74bf89a5c97ac7ae10c43, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679831685314 2023-03-26 11:54:46,377 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into 35d2e192ceb54ce8a63a4c46693e60e1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:46,377 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,377 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=12, startTime=1679831686298; duration=0sec 2023-03-26 11:54:46,377 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:46,399 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,402 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#265 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:46,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,403 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,408 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:46,408 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,409 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,412 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,412 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,412 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,412 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 11:54:46,412 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:46,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:46,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:46,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:46,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:46,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:46,424 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/493211f4dc494247a275a80cba67b190 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/493211f4dc494247a275a80cba67b190 2023-03-26 11:54:46,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 493211f4dc494247a275a80cba67b190(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:46,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=12, startTime=1679831686299; duration=0sec 2023-03-26 11:54:46,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:46,432 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/1292ddf0e52c42c9ac199daff9af33f1 2023-03-26 11:54:46,445 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/726270667f19417cae80d145ee77ba2e 2023-03-26 11:54:46,462 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=301 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/ec9bc392c07d4ea1ab03772d05d8ea17 2023-03-26 11:54:46,466 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/1292ddf0e52c42c9ac199daff9af33f1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1292ddf0e52c42c9ac199daff9af33f1 2023-03-26 11:54:46,470 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1292ddf0e52c42c9ac199daff9af33f1, entries=150, sequenceid=301, filesize=11.9 K 2023-03-26 11:54:46,471 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/726270667f19417cae80d145ee77ba2e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/726270667f19417cae80d145ee77ba2e 2023-03-26 11:54:46,474 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/726270667f19417cae80d145ee77ba2e, entries=150, sequenceid=301, filesize=11.9 K 2023-03-26 11:54:46,475 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/ec9bc392c07d4ea1ab03772d05d8ea17 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ec9bc392c07d4ea1ab03772d05d8ea17 2023-03-26 11:54:46,487 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ec9bc392c07d4ea1ab03772d05d8ea17, entries=150, sequenceid=301, filesize=11.9 K 2023-03-26 11:54:46,488 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 70cba8f2e9bfa903d814db3e7b9776da in 76ms, sequenceid=301, compaction requested=false 2023-03-26 11:54:46,488 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,488 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,494 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7909b9f7[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,496 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,499 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,499 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,501 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,501 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,502 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,503 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,508 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,508 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,567 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:46,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:46,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:46,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:46,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:46,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:46,568 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:46,568 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:46,583 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/da0034299ce842a085e0d9f47741cbe0 2023-03-26 11:54:46,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831686586, max timestamp=1679831686598Num uniques 859; 2023-03-26 11:54:46,602 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:46,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831686586, max timestamp=1679831686598Num uniques 859; 2023-03-26 11:54:46,605 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:54:46,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831686586, max timestamp=1679831686598Num uniques 859; 2023-03-26 11:54:46,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:54:46,609 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,609 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,611 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,628 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/a0029b4ef86443f4a0e56f5ce3c3deda 2023-03-26 11:54:46,629 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,629 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831746627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,631 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,633 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,633 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,633 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,633 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,634 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831746630, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,634 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:46,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831746629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,635 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:46,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,653 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,656 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38d109ea[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,659 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,659 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,660 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,660 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,660 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,661 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,661 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,661 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,661 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:46,666 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,666 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,666 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,677 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,687 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,687 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,687 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:46,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,688 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,688 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,694 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=313 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/9f8e8b4b4d69492f88c8de5b60f08046 2023-03-26 11:54:46,699 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/da0034299ce842a085e0d9f47741cbe0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/da0034299ce842a085e0d9f47741cbe0 2023-03-26 11:54:46,710 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/da0034299ce842a085e0d9f47741cbe0, entries=200, sequenceid=313, filesize=14.3 K 2023-03-26 11:54:46,710 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/a0029b4ef86443f4a0e56f5ce3c3deda as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a0029b4ef86443f4a0e56f5ce3c3deda 2023-03-26 11:54:46,714 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a0029b4ef86443f4a0e56f5ce3c3deda, entries=150, sequenceid=313, filesize=11.9 K 2023-03-26 11:54:46,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/9f8e8b4b4d69492f88c8de5b60f08046 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9f8e8b4b4d69492f88c8de5b60f08046 2023-03-26 11:54:46,729 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9f8e8b4b4d69492f88c8de5b60f08046, entries=150, sequenceid=313, filesize=11.9 K 2023-03-26 11:54:46,730 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=128.92 KB/132010 for 70cba8f2e9bfa903d814db3e7b9776da in 163ms, sequenceid=313, compaction requested=true 2023-03-26 11:54:46,730 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:46,730 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:46,730 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:46,730 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:46,731 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:46,732 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39646 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:46,732 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:46,732 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:46,732 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,732 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:46,732 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/61efb34b34b544deb86d0f79e6c04017, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1292ddf0e52c42c9ac199daff9af33f1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/da0034299ce842a085e0d9f47741cbe0] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=38.7 K 2023-03-26 11:54:46,732 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,732 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/35d2e192ceb54ce8a63a4c46693e60e1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/726270667f19417cae80d145ee77ba2e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a0029b4ef86443f4a0e56f5ce3c3deda] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.3 K 2023-03-26 11:54:46,732 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 61efb34b34b544deb86d0f79e6c04017, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679831685314 2023-03-26 11:54:46,733 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 35d2e192ceb54ce8a63a4c46693e60e1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679831685314 2023-03-26 11:54:46,733 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1292ddf0e52c42c9ac199daff9af33f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679831685435 2023-03-26 11:54:46,733 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 726270667f19417cae80d145ee77ba2e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679831685435 2023-03-26 11:54:46,733 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting da0034299ce842a085e0d9f47741cbe0, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679831686561 2023-03-26 11:54:46,734 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a0029b4ef86443f4a0e56f5ce3c3deda, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679831686561 2023-03-26 11:54:46,742 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:46,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:46,743 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=135.63 KB heapSize=388.23 KB 2023-03-26 11:54:46,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:46,744 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 11:54:46,744 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:46,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 11:54:46,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:46,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-03-26 11:54:46,749 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#273 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:46,757 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831746753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,759 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831746755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831746755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,788 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/288ddb15be994538979e5d51cba11c45 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/288ddb15be994538979e5d51cba11c45 2023-03-26 11:54:46,788 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,791 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/c3b8b7691ada4fdfb9c4e2f7841a0386 2023-03-26 11:54:46,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,794 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 288ddb15be994538979e5d51cba11c45(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:46,794 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:46,794 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=13, startTime=1679831686730; duration=0sec 2023-03-26 11:54:46,794 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:46,794 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:46,795 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37206 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:46,795 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:46,795 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,795 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/493211f4dc494247a275a80cba67b190, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ec9bc392c07d4ea1ab03772d05d8ea17, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9f8e8b4b4d69492f88c8de5b60f08046] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.3 K 2023-03-26 11:54:46,796 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 493211f4dc494247a275a80cba67b190, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=282, earliestPutTs=1679831685314 2023-03-26 11:54:46,796 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ec9bc392c07d4ea1ab03772d05d8ea17, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=301, earliestPutTs=1679831685435 2023-03-26 11:54:46,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,797 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9f8e8b4b4d69492f88c8de5b60f08046, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679831686561 2023-03-26 11:54:46,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,797 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,799 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,813 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#276 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:46,820 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:46,820 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37496 deadline: 1679831746825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,832 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,832 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,832 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,833 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,833 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:46,833 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831746830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,835 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,835 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,835 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,835 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,837 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7beef90f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,839 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,839 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,841 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,842 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,843 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,843 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,844 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,856 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:46,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,857 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,857 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,857 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,858 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831746859, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831746860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,862 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:46,862 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831746861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,864 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/d45ff69737974986b6072a005082d8c1 2023-03-26 11:54:46,957 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:46,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:46,959 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:46,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:46,960 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:46,960 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:46,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:46,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:46,961 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,961 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,962 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,962 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,962 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,962 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:46,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,963 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:46,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:46,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:46,963 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,963 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:46,963 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:46,963 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:46,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:46,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:46,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:46,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:46,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7126bd5f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:46,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:46,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:46,970 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:46,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:46,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:46,973 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:46,973 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:46,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:46,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:46,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:46,973 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:46,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:46,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:46,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:46,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:46,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:46,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:46,984 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:46,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:46,985 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:46,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:46,985 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:46,985 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:47,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831747063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,064 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831747063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831747064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,085 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,089 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,091 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,093 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,093 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,093 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,093 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,094 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,094 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,095 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,095 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45d4748d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,102 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,104 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,105 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,105 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,108 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,108 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,108 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,113 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,113 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,113 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,113 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,114 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,114 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,114 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,114 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,114 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,114 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,114 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,170 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/a23f768994af4f08be23845b066a08bb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a23f768994af4f08be23845b066a08bb 2023-03-26 11:54:47,174 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into a23f768994af4f08be23845b066a08bb(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:47,175 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:47,175 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=13, startTime=1679831686730; duration=0sec 2023-03-26 11:54:47,175 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:47,214 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,216 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,218 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,218 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,218 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,218 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,221 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,221 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,226 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,227 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,227 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,227 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,227 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,228 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,230 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,231 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75fabb84[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,234 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,234 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,235 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,236 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,236 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,243 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,243 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,246 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,246 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,246 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,253 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,253 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,253 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:47,279 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/99aa04d29bae443195b822cce5864555 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/99aa04d29bae443195b822cce5864555 2023-03-26 11:54:47,284 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 99aa04d29bae443195b822cce5864555(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:47,284 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:47,284 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=13, startTime=1679831686731; duration=0sec 2023-03-26 11:54:47,285 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:47,325 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.57 KB at sequenceid=351 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/896756b1926f40358ed235a22ecfbb0c 2023-03-26 11:54:47,329 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/c3b8b7691ada4fdfb9c4e2f7841a0386 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c3b8b7691ada4fdfb9c4e2f7841a0386 2023-03-26 11:54:47,333 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c3b8b7691ada4fdfb9c4e2f7841a0386, entries=200, sequenceid=351, filesize=14.3 K 2023-03-26 11:54:47,334 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/d45ff69737974986b6072a005082d8c1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d45ff69737974986b6072a005082d8c1 2023-03-26 11:54:47,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d45ff69737974986b6072a005082d8c1, entries=150, sequenceid=351, filesize=11.9 K 2023-03-26 11:54:47,340 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/896756b1926f40358ed235a22ecfbb0c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/896756b1926f40358ed235a22ecfbb0c 2023-03-26 11:54:47,353 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,354 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/896756b1926f40358ed235a22ecfbb0c, entries=150, sequenceid=351, filesize=11.9 K 2023-03-26 11:54:47,356 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~142.33 KB/145750, heapSize ~408.81 KB/418624, currentSize=40.25 KB/41220 for 70cba8f2e9bfa903d814db3e7b9776da in 613ms, sequenceid=351, compaction requested=false 2023-03-26 11:54:47,356 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:47,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:47,367 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:54:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:47,368 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,369 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,369 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,369 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,369 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,376 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,385 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,385 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,385 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,385 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,386 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,388 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,390 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,391 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@79d002b5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,393 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831687382, max timestamp=1679831687393Num uniques 859; 2023-03-26 11:54:47,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:54:47,396 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831687382, max timestamp=1679831687393Num uniques 859; 2023-03-26 11:54:47,397 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:54:47,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831687382, max timestamp=1679831687393Num uniques 859; 2023-03-26 11:54:47,398 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,402 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,402 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,407 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,410 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:47,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,410 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,410 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,418 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831747415, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,419 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831747417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,421 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831747419, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,510 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,514 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,515 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,517 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,517 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,517 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,517 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,518 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,521 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,521 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831747519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,522 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831747520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d589495[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831747522, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,524 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,529 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,529 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,530 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,531 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,537 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,537 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,637 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,640 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,641 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,642 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,642 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,642 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,642 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,643 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,643 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,643 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,644 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,645 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,645 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,646 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bceefaf[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,648 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,649 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,649 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,651 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,653 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,653 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,657 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,664 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,664 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,665 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,665 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831747723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,724 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831747723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,727 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:47,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831747725, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,765 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,768 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,768 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,768 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,768 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,773 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,774 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bc859c9[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,783 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,787 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/c011ea29433e4ed5a8b76db9258fc383 2023-03-26 11:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,789 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,789 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,805 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,805 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,805 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,905 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:47,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:47,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:47,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:47,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:47,908 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:47,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:47,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:47,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,909 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,910 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,911 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:47,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:47,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:47,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:47,912 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,912 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:47,912 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:47,912 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:47,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:47,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:47,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27439af[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:47,915 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:47,916 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,917 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:47,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:47,917 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,917 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:47,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:47,919 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:47,919 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,919 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,919 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:47,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:47,921 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:47,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:47,922 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:47,922 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:47,922 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:47,923 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,923 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:47,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,923 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:47,923 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:47,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:47,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:47,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:47,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:47,927 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:47,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:47,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:47,927 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:47,927 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:47,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,027 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831748025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,027 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:48,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:48,028 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831748026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:48,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:48,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:48,030 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:48,031 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,031 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:48,031 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:48,031 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,031 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831748029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,031 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,033 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,034 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:48,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:48,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:48,035 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:48,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf1f948[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:48,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:48,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,040 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:48,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:48,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:48,042 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:48,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:48,044 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:48,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:48,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:48,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:48,044 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:48,046 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,046 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,046 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,048 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:48,051 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:48,051 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,051 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,052 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,152 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:48,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:48,154 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:48,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:48,155 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:48,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:48,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:48,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,156 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:48,157 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:48,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,158 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:48,158 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:48,158 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:48,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:48,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:48,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:48,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:48,160 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,160 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,161 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:48,162 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@576ed20e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:48,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:48,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,163 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:48,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:48,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:48,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,168 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:48,171 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:48,171 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:48,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:48,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:48,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:48,171 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:48,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,173 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,174 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,177 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:48,177 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,177 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:48,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,177 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,177 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:48,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,199 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/bd2bf18760a2434da98c1cb6082a8eca 2023-03-26 11:54:48,277 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:48,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:48,279 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:48,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:48,280 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:48,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:48,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:48,281 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,281 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,282 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:48,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:48,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,283 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:48,283 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:48,283 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:48,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:48,284 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,284 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:48,284 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:48,284 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:48,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:48,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,286 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:48,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:48,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e51cf3b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:48,288 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:48,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:48,289 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,289 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:48,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,291 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:48,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:48,294 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:48,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:48,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:48,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:48,294 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:48,295 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,295 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,295 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,296 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,297 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:48,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:48,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,299 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,300 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:48,300 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:48,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,300 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:48,300 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,400 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:48,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:48,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:48,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:48,404 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:48,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:48,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:48,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:48,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:48,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,407 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:48,407 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:48,407 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:48,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:48,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:48,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:48,407 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:48,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:48,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:48,408 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:48,408 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:48,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,411 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,411 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:48,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:48,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d4b3a1f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:48,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:48,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,426 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:48,426 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,426 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:48,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:48,428 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,428 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:48,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:48,431 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:48,431 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:48,431 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:48,431 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:48,431 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:48,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,433 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,437 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,437 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,437 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:48,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:48,438 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,438 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,438 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,438 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,438 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,438 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:48,438 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:48,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:48,438 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,529 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831748528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831748532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831748533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,538 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:48,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:48,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:48,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:48,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:48,541 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:48,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:48,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:48,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,543 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,543 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,544 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,544 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:48,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,544 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,545 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:48,545 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:48,545 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:48,545 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,545 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:48,545 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:48,545 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,545 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:48,546 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:48,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,547 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,547 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,547 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:48,549 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3984a147[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:48,549 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:48,549 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:48,550 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,550 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:48,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:48,552 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,552 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:48,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:48,555 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:48,555 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:48,555 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:48,555 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:48,555 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:48,558 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,558 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,558 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:48,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,559 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,561 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:48,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:48,563 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,563 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:48,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:48,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:48,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,564 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:48,564 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,564 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:48,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:48,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:48,610 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=364 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/750f2bbcafb349f7b92945b435b8d97b 2023-03-26 11:54:48,615 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/c011ea29433e4ed5a8b76db9258fc383 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c011ea29433e4ed5a8b76db9258fc383 2023-03-26 11:54:48,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c011ea29433e4ed5a8b76db9258fc383, entries=150, sequenceid=364, filesize=11.9 K 2023-03-26 11:54:48,621 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/bd2bf18760a2434da98c1cb6082a8eca as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/bd2bf18760a2434da98c1cb6082a8eca 2023-03-26 11:54:48,626 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/bd2bf18760a2434da98c1cb6082a8eca, entries=150, sequenceid=364, filesize=11.9 K 2023-03-26 11:54:48,627 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/750f2bbcafb349f7b92945b435b8d97b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/750f2bbcafb349f7b92945b435b8d97b 2023-03-26 11:54:48,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/750f2bbcafb349f7b92945b435b8d97b, entries=150, sequenceid=364, filesize=11.9 K 2023-03-26 11:54:48,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=135.63 KB/138880 for 70cba8f2e9bfa903d814db3e7b9776da in 1270ms, sequenceid=364, compaction requested=true 2023-03-26 11:54:48,637 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:48,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:48,637 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:48,637 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:48,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:48,637 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:48,638 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39748 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:48,638 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:48,638 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:48,638 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:48,638 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,638 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,638 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/288ddb15be994538979e5d51cba11c45, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c3b8b7691ada4fdfb9c4e2f7841a0386, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c011ea29433e4ed5a8b76db9258fc383] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=38.8 K 2023-03-26 11:54:48,638 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a23f768994af4f08be23845b066a08bb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d45ff69737974986b6072a005082d8c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/bd2bf18760a2434da98c1cb6082a8eca] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.4 K 2023-03-26 11:54:48,639 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 288ddb15be994538979e5d51cba11c45, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679831686561 2023-03-26 11:54:48,639 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a23f768994af4f08be23845b066a08bb, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679831686561 2023-03-26 11:54:48,639 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c3b8b7691ada4fdfb9c4e2f7841a0386, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679831686627 2023-03-26 11:54:48,639 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d45ff69737974986b6072a005082d8c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679831686627 2023-03-26 11:54:48,639 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c011ea29433e4ed5a8b76db9258fc383, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679831686751 2023-03-26 11:54:48,639 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bd2bf18760a2434da98c1cb6082a8eca, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679831686751 2023-03-26 11:54:48,649 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:48,652 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:48,664 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:48,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:48,666 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:48,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:48,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:48,676 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:48,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:48,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:48,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,676 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:48,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:48,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:48,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:48,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:48,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:48,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:48,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:48,681 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:48,681 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:48,681 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:48,681 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=135.63 KB heapSize=388.23 KB 2023-03-26 11:54:48,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:48,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:48,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:48,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:48,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:48,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:48,684 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/043dc9c1f3584ae8a0f3a11f21848d64 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/043dc9c1f3584ae8a0f3a11f21848d64 2023-03-26 11:54:48,684 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/aea125ad245f47ddac799c61a6b68eb1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/aea125ad245f47ddac799c61a6b68eb1 2023-03-26 11:54:48,691 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into aea125ad245f47ddac799c61a6b68eb1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:48,691 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:48,691 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=13, startTime=1679831688637; duration=0sec 2023-03-26 11:54:48,691 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:48,691 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:48,691 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 043dc9c1f3584ae8a0f3a11f21848d64(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:48,691 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:48,692 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=13, startTime=1679831688637; duration=0sec 2023-03-26 11:54:48,692 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:48,692 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37308 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:48,692 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:48,692 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:48,692 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/99aa04d29bae443195b822cce5864555, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/896756b1926f40358ed235a22ecfbb0c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/750f2bbcafb349f7b92945b435b8d97b] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.4 K 2023-03-26 11:54:48,696 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 99aa04d29bae443195b822cce5864555, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=313, earliestPutTs=1679831686561 2023-03-26 11:54:48,696 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 896756b1926f40358ed235a22ecfbb0c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=351, earliestPutTs=1679831686627 2023-03-26 11:54:48,697 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 750f2bbcafb349f7b92945b435b8d97b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679831686751 2023-03-26 11:54:48,705 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#284 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:48,717 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/322478c7f58143c4a4130ae7ff1e4207 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/322478c7f58143c4a4130ae7ff1e4207 2023-03-26 11:54:48,721 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 322478c7f58143c4a4130ae7ff1e4207(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:48,721 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:48,721 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=13, startTime=1679831688637; duration=0sec 2023-03-26 11:54:48,721 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:48,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:48,837 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:48,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831748850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,853 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831748851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831748954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:48,955 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:48,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831748954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,105 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/c38cb8526c014bbf93a90e3650933a5c 2023-03-26 11:54:49,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:37552 deadline: 1679831749155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831749157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831749460, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,461 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831749461, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,517 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/80154b11b1314b768085dc3ff453d607 2023-03-26 11:54:49,534 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831749533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831749537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,544 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:49,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831749543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,929 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/8327754de4b54561b4770f5267cbce78 2023-03-26 11:54:49,933 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/c38cb8526c014bbf93a90e3650933a5c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c38cb8526c014bbf93a90e3650933a5c 2023-03-26 11:54:49,937 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c38cb8526c014bbf93a90e3650933a5c, entries=150, sequenceid=401, filesize=11.9 K 2023-03-26 11:54:49,938 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/80154b11b1314b768085dc3ff453d607 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/80154b11b1314b768085dc3ff453d607 2023-03-26 11:54:49,942 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/80154b11b1314b768085dc3ff453d607, entries=150, sequenceid=401, filesize=11.9 K 2023-03-26 11:54:49,943 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/8327754de4b54561b4770f5267cbce78 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8327754de4b54561b4770f5267cbce78 2023-03-26 11:54:49,946 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8327754de4b54561b4770f5267cbce78, entries=150, sequenceid=401, filesize=11.9 K 2023-03-26 11:54:49,947 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~135.63 KB/138880, heapSize ~388.19 KB/397504, currentSize=46.96 KB/48090 for 70cba8f2e9bfa903d814db3e7b9776da in 1266ms, sequenceid=401, compaction requested=false 2023-03-26 11:54:49,947 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:49,947 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,947 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:49,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:49,949 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,949 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:49,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:49,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:49,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:49,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@185bfadb[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:49,951 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:49,951 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,953 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,953 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:49,953 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:49,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:49,955 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,955 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:49,955 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:49,955 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:49,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:49,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:49,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:49,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:49,959 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:49,959 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:49,959 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:49,959 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:49,959 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:49,959 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:49,961 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,961 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:49,961 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:49,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:49,961 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:49,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:49,963 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:49,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:49,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:49,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:49,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:49,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:49,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:49,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:49,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:49,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:49,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:54:49,966 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:49,966 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:49,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:49,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:49,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:49,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:49,966 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:49,966 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:49,965 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:49,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:49,967 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,967 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:49,967 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:49,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:49,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:49,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:49,973 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/2f8ed318aab74dffa67d0e2a1910f755 2023-03-26 11:54:49,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831689983, max timestamp=1679831689993Num uniques 859; 2023-03-26 11:54:49,995 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:49,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831689983, max timestamp=1679831689993Num uniques 859; 2023-03-26 11:54:49,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831689983, max timestamp=1679831689993Num uniques 859; 2023-03-26 11:54:49,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:49,996 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:54:50,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831750020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831750022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,066 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:50,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:50,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:50,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:50,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:50,070 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:50,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:50,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:50,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,073 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:50,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,073 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,074 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:50,074 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:50,074 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:50,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:50,075 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,075 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:50,075 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:50,075 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:50,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:50,076 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,077 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:50,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:50,078 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7893c6fc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:50,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:50,079 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:50,080 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:50,080 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,080 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,081 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:50,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:50,084 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:50,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:50,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:50,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:50,084 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:50,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,086 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,090 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,090 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,090 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,090 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:50,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:50,090 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,090 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,091 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,091 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,091 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:50,091 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,091 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:50,091 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:50,126 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831750125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831750125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,191 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:50,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:50,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:50,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:50,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:50,196 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:50,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:50,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:50,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:50,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,199 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:50,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:50,199 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:50,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:50,200 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,200 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:50,200 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:50,200 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:50,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:50,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:50,203 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9c8033[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:50,203 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:50,203 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:50,206 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:50,207 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:50,207 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,207 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:50,208 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,208 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:50,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:50,211 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:50,211 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:50,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:50,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:50,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:50,211 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:50,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,213 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,213 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:50,226 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,226 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,226 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,226 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,227 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,326 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:50,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:50,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:50,329 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831750328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,330 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831750328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:50,330 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:50,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:50,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:50,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:50,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,333 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,333 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:50,333 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:50,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:50,334 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,334 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:50,334 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:50,334 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:50,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:50,336 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,336 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:50,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@700e9a7e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:50,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:50,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,338 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:50,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:50,339 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:50,339 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,339 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:50,341 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,341 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:50,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:50,343 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:50,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:50,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:50,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:50,343 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:50,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,346 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:50,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,348 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:50,348 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/1ed890fbecec4f50837b8d5f5973fc90 2023-03-26 11:54:50,396 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=415 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/4b7c5af9419540a88fa9c122e73efd60 2023-03-26 11:54:50,399 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/2f8ed318aab74dffa67d0e2a1910f755 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/2f8ed318aab74dffa67d0e2a1910f755 2023-03-26 11:54:50,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/2f8ed318aab74dffa67d0e2a1910f755, entries=150, sequenceid=415, filesize=11.9 K 2023-03-26 11:54:50,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/1ed890fbecec4f50837b8d5f5973fc90 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/1ed890fbecec4f50837b8d5f5973fc90 2023-03-26 11:54:50,413 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/1ed890fbecec4f50837b8d5f5973fc90, entries=150, sequenceid=415, filesize=11.9 K 2023-03-26 11:54:50,413 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/4b7c5af9419540a88fa9c122e73efd60 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/4b7c5af9419540a88fa9c122e73efd60 2023-03-26 11:54:50,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/4b7c5af9419540a88fa9c122e73efd60, entries=150, sequenceid=415, filesize=11.9 K 2023-03-26 11:54:50,424 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 70cba8f2e9bfa903d814db3e7b9776da in 458ms, sequenceid=415, compaction requested=true 2023-03-26 11:54:50,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:50,424 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:50,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:50,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:50,425 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 11:54:50,425 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:50,428 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:50,428 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:50,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/043dc9c1f3584ae8a0f3a11f21848d64, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c38cb8526c014bbf93a90e3650933a5c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/2f8ed318aab74dffa67d0e2a1910f755] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.5 K 2023-03-26 11:54:50,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:50,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:50,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/aea125ad245f47ddac799c61a6b68eb1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/80154b11b1314b768085dc3ff453d607, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/1ed890fbecec4f50837b8d5f5973fc90] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.5 K 2023-03-26 11:54:50,429 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 043dc9c1f3584ae8a0f3a11f21848d64, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679831686751 2023-03-26 11:54:50,431 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c38cb8526c014bbf93a90e3650933a5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679831687414 2023-03-26 11:54:50,431 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting aea125ad245f47ddac799c61a6b68eb1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679831686751 2023-03-26 11:54:50,431 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2f8ed318aab74dffa67d0e2a1910f755, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679831688838 2023-03-26 11:54:50,432 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 80154b11b1314b768085dc3ff453d607, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679831687414 2023-03-26 11:54:50,433 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1ed890fbecec4f50837b8d5f5973fc90, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679831688838 2023-03-26 11:54:50,442 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#290 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:50,445 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:50,448 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:50,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:50,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:50,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:50,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:50,468 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:50,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:50,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:50,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,471 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,471 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,472 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:50,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,472 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:50,476 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:50,476 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:50,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:50,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:50,476 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 11:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:50,477 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:50,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:50,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:50,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831750656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831750658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,762 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831750759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:50,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831750763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,874 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/48ba7fcb10fb405aa5128888803e640d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/48ba7fcb10fb405aa5128888803e640d 2023-03-26 11:54:50,879 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/6208354f094d4807a91596f4760c0f8a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/6208354f094d4807a91596f4760c0f8a 2023-03-26 11:54:50,880 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 48ba7fcb10fb405aa5128888803e640d(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:50,880 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:50,880 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=13, startTime=1679831690424; duration=0sec 2023-03-26 11:54:50,880 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:50,880 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:50,882 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:50,882 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:50,882 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,882 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/322478c7f58143c4a4130ae7ff1e4207, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8327754de4b54561b4770f5267cbce78, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/4b7c5af9419540a88fa9c122e73efd60] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.5 K 2023-03-26 11:54:50,882 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 322478c7f58143c4a4130ae7ff1e4207, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=364, earliestPutTs=1679831686751 2023-03-26 11:54:50,883 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8327754de4b54561b4770f5267cbce78, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1679831687414 2023-03-26 11:54:50,883 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 4b7c5af9419540a88fa9c122e73efd60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679831688838 2023-03-26 11:54:50,884 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into 6208354f094d4807a91596f4760c0f8a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:50,884 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:50,884 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=13, startTime=1679831690425; duration=0sec 2023-03-26 11:54:50,884 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:50,884 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/7bc26f2e30b14b4caf8335f48c8f3bcd 2023-03-26 11:54:50,891 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#293 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:50,908 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/1143f4a0c9f843e4894bfc2868e5b0cd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1143f4a0c9f843e4894bfc2868e5b0cd 2023-03-26 11:54:50,913 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 1143f4a0c9f843e4894bfc2868e5b0cd(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:50,913 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:50,913 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=13, startTime=1679831690425; duration=0sec 2023-03-26 11:54:50,913 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:50,918 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/9e03e8226e5e431fbda59c630e978ca3 2023-03-26 11:54:50,928 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=451 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/d6ec087222004c7caedb0a57d06dba3e 2023-03-26 11:54:50,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/7bc26f2e30b14b4caf8335f48c8f3bcd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/7bc26f2e30b14b4caf8335f48c8f3bcd 2023-03-26 11:54:50,936 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/7bc26f2e30b14b4caf8335f48c8f3bcd, entries=150, sequenceid=451, filesize=11.9 K 2023-03-26 11:54:50,937 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/9e03e8226e5e431fbda59c630e978ca3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e03e8226e5e431fbda59c630e978ca3 2023-03-26 11:54:50,941 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e03e8226e5e431fbda59c630e978ca3, entries=150, sequenceid=451, filesize=11.9 K 2023-03-26 11:54:50,942 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/d6ec087222004c7caedb0a57d06dba3e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d6ec087222004c7caedb0a57d06dba3e 2023-03-26 11:54:50,947 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d6ec087222004c7caedb0a57d06dba3e, entries=150, sequenceid=451, filesize=11.9 K 2023-03-26 11:54:50,948 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 70cba8f2e9bfa903d814db3e7b9776da in 472ms, sequenceid=451, compaction requested=false 2023-03-26 11:54:50,948 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:50,948 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,950 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,951 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:50,952 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37443b86[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:50,952 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:50,953 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,954 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,954 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:50,954 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:50,954 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:50,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:50,957 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:50,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:50,960 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:50,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:50,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:50,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:50,960 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:50,962 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,962 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:50,962 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:50,963 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:50,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:50,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:50,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:50,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:50,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:50,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:50,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:50,968 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:50,968 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:50,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:50,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:50,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:50,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:50,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:50,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:50,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:50,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:50,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:50,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:50,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:50,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:50,969 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:50,969 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:50,999 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/63e1c360921644aaa9395380f221d9be 2023-03-26 11:54:51,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,001 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831750997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831750999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,022 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/7579ba9dd4564eec84047a34e3777f5f 2023-03-26 11:54:51,069 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,071 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,072 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,074 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,075 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,075 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,075 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,076 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,076 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,076 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,076 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,078 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,079 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,080 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5186afb4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,080 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,080 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,081 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,081 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,081 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,083 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,083 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,085 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,085 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,085 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,087 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,087 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,087 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,092 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,092 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:51,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831751102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831751103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,192 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,196 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,201 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,201 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,203 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,203 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,204 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,204 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,204 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,206 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,206 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,207 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,208 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@599330af[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,210 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,210 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,210 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,210 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,212 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,212 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,212 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,214 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,214 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,214 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,215 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,216 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,216 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,216 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,217 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,218 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,221 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,221 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,221 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,305 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831751304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,308 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831751307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,321 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,324 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,326 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,326 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,327 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,327 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,327 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,327 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,329 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d4a8c89[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,333 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,333 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,333 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,335 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,335 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,335 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,337 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,337 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,341 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:51,344 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,344 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,345 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,440 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=467 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/1e54ee9f9165495f81d0507e5a02c7aa 2023-03-26 11:54:51,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/63e1c360921644aaa9395380f221d9be as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/63e1c360921644aaa9395380f221d9be 2023-03-26 11:54:51,445 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,448 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,448 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/63e1c360921644aaa9395380f221d9be, entries=150, sequenceid=467, filesize=11.9 K 2023-03-26 11:54:51,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,449 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/7579ba9dd4564eec84047a34e3777f5f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/7579ba9dd4564eec84047a34e3777f5f 2023-03-26 11:54:51,449 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,451 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,453 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,453 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b4414ad[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,454 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,455 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,456 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,457 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/7579ba9dd4564eec84047a34e3777f5f, entries=150, sequenceid=467, filesize=11.9 K 2023-03-26 11:54:51,458 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/1e54ee9f9165495f81d0507e5a02c7aa as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1e54ee9f9165495f81d0507e5a02c7aa 2023-03-26 11:54:51,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,461 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,461 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,461 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1e54ee9f9165495f81d0507e5a02c7aa, entries=150, sequenceid=467, filesize=11.9 K 2023-03-26 11:54:51,462 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 70cba8f2e9bfa903d814db3e7b9776da in 496ms, sequenceid=467, compaction requested=true 2023-03-26 11:54:51,462 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:51,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:51,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:51,462 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:51,462 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:51,462 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:51,463 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,463 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,463 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,464 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:51,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:51,464 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:51,464 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:51,464 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,464 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/6208354f094d4807a91596f4760c0f8a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e03e8226e5e431fbda59c630e978ca3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/7579ba9dd4564eec84047a34e3777f5f] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.6 K 2023-03-26 11:54:51,464 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,464 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/48ba7fcb10fb405aa5128888803e640d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/7bc26f2e30b14b4caf8335f48c8f3bcd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/63e1c360921644aaa9395380f221d9be] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.6 K 2023-03-26 11:54:51,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6208354f094d4807a91596f4760c0f8a, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679831688838 2023-03-26 11:54:51,465 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 48ba7fcb10fb405aa5128888803e640d, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679831688838 2023-03-26 11:54:51,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,465 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9e03e8226e5e431fbda59c630e978ca3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679831690012 2023-03-26 11:54:51,465 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 7bc26f2e30b14b4caf8335f48c8f3bcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679831690012 2023-03-26 11:54:51,465 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 7579ba9dd4564eec84047a34e3777f5f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1679831690656 2023-03-26 11:54:51,465 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 63e1c360921644aaa9395380f221d9be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1679831690656 2023-03-26 11:54:51,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,472 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#299 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:51,472 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#300 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:51,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,473 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,473 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,473 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,473 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,474 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,474 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,474 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,489 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/829398fe735741e98a74c19eae5a74e6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/829398fe735741e98a74c19eae5a74e6 2023-03-26 11:54:51,493 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 829398fe735741e98a74c19eae5a74e6(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:51,493 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:51,493 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=13, startTime=1679831691462; duration=0sec 2023-03-26 11:54:51,493 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:51,493 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:54:51,495 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37512 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:54:51,495 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:51,495 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,495 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1143f4a0c9f843e4894bfc2868e5b0cd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d6ec087222004c7caedb0a57d06dba3e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1e54ee9f9165495f81d0507e5a02c7aa] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=36.6 K 2023-03-26 11:54:51,495 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1143f4a0c9f843e4894bfc2868e5b0cd, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=415, earliestPutTs=1679831688838 2023-03-26 11:54:51,496 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d6ec087222004c7caedb0a57d06dba3e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=451, earliestPutTs=1679831690012 2023-03-26 11:54:51,496 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1e54ee9f9165495f81d0507e5a02c7aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1679831690656 2023-03-26 11:54:51,502 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:51,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:51,552 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:54:51,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:51,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:51,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:51,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:51,552 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:51,552 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:51,559 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/89e173fab6b84e418304b94f48ea757c 2023-03-26 11:54:51,569 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831751568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,572 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831751569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831751569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,574 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,576 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,577 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,577 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,579 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,580 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,580 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,580 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,581 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,581 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,581 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,581 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f5bf97e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,587 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,587 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,588 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,588 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,588 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,591 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,591 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,594 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,594 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,596 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,596 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,596 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,602 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,602 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,602 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,602 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831751606, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,608 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,610 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831751609, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831751670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,674 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831751673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831751673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,702 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,704 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,706 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,710 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,710 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,710 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,711 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,711 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,711 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,711 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,711 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,713 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f8da272[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,715 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,715 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,716 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,716 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,716 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,716 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,718 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,718 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,720 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,720 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,722 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,722 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,722 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,724 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,727 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,727 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,727 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,727 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:51,727 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,727 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,827 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,829 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,831 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,833 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,833 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,833 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,834 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,834 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,834 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,834 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,834 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,834 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,834 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,837 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66aa55ce[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,839 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,839 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,840 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,840 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,840 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,841 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,843 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,844 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,845 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,845 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,845 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,847 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,850 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:51,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,851 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,851 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:51,875 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831751873, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831751876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,876 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:51,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831751876, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,888 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/074ece6baa104e37bc708a9c8af82d4e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/074ece6baa104e37bc708a9c8af82d4e 2023-03-26 11:54:51,893 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into 074ece6baa104e37bc708a9c8af82d4e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:51,893 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:51,893 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=13, startTime=1679831691462; duration=0sec 2023-03-26 11:54:51,893 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:51,912 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/9079b852613d491da98b42d9784739b4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9079b852613d491da98b42d9784739b4 2023-03-26 11:54:51,917 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 9079b852613d491da98b42d9784739b4(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:51,917 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:51,917 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=13, startTime=1679831691462; duration=0sec 2023-03-26 11:54:51,917 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:51,951 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:51,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:51,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:51,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:51,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:51,954 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:51,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:51,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:51,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,954 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,955 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,955 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,956 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,956 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,956 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:51,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,957 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:51,957 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:51,957 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:51,957 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,957 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:51,957 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:51,957 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:51,957 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,959 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:51,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:51,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@da0d4ca[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:51,962 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:51,962 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:51,962 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,962 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:51,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:51,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:51,968 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:51,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:51,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:51,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:51,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:51,969 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:51,970 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,970 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/fc2a2adae6b64235b4ac6a5201b61825 2023-03-26 11:54:51,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:51,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:51,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:51,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:51,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:51,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:51,976 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:51,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:51,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:51,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:51,976 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:51,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:51,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:51,980 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:51,980 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,080 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,083 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,086 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,086 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,086 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:52,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,089 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,089 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,090 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dc04c09[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,092 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,095 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,096 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,097 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,097 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,097 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,106 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,109 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,106 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:52,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,109 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,109 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831752111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,113 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831752113, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,179 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831752178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831752178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,180 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831752178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,210 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,221 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,224 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,225 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,226 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,227 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,227 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,228 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,228 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,228 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,228 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,229 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,229 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:52,229 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,231 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,231 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,232 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fdcfff1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,234 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,237 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,239 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,239 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,239 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,239 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,239 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,241 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,241 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,241 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,241 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,245 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,245 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,245 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,245 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,245 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,245 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,346 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,347 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,349 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,349 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,350 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,351 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,351 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,351 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,352 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,352 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:52,352 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,355 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b6cd1e1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,357 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,359 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,362 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,362 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,362 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,362 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,362 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,366 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,371 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,371 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,370 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:52,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,394 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=488 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/6f575164fc76450e9dd4cb4f94460bcd 2023-03-26 11:54:52,401 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/89e173fab6b84e418304b94f48ea757c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/89e173fab6b84e418304b94f48ea757c 2023-03-26 11:54:52,406 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/89e173fab6b84e418304b94f48ea757c, entries=200, sequenceid=488, filesize=14.3 K 2023-03-26 11:54:52,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/fc2a2adae6b64235b4ac6a5201b61825 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fc2a2adae6b64235b4ac6a5201b61825 2023-03-26 11:54:52,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fc2a2adae6b64235b4ac6a5201b61825, entries=150, sequenceid=488, filesize=11.9 K 2023-03-26 11:54:52,411 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/6f575164fc76450e9dd4cb4f94460bcd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f575164fc76450e9dd4cb4f94460bcd 2023-03-26 11:54:52,415 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f575164fc76450e9dd4cb4f94460bcd, entries=150, sequenceid=488, filesize=11.9 K 2023-03-26 11:54:52,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 70cba8f2e9bfa903d814db3e7b9776da in 863ms, sequenceid=488, compaction requested=false 2023-03-26 11:54:52,416 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:52,471 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,474 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,474 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,476 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,476 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,476 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,476 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,477 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,477 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,477 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,477 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,478 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:52,478 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:52,486 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/500c73f9988243e591e9f2cec2ecd3e5 2023-03-26 11:54:52,495 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/fdffe2c0855948c7b8303085732d3842 2023-03-26 11:54:52,508 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=502 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/243d7ff953e243ef84c132ac27b97931 2023-03-26 11:54:52,515 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/500c73f9988243e591e9f2cec2ecd3e5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/500c73f9988243e591e9f2cec2ecd3e5 2023-03-26 11:54:52,520 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/500c73f9988243e591e9f2cec2ecd3e5, entries=150, sequenceid=502, filesize=11.9 K 2023-03-26 11:54:52,520 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/fdffe2c0855948c7b8303085732d3842 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fdffe2c0855948c7b8303085732d3842 2023-03-26 11:54:52,524 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fdffe2c0855948c7b8303085732d3842, entries=150, sequenceid=502, filesize=11.9 K 2023-03-26 11:54:52,525 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/243d7ff953e243ef84c132ac27b97931 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/243d7ff953e243ef84c132ac27b97931 2023-03-26 11:54:52,529 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/243d7ff953e243ef84c132ac27b97931, entries=150, sequenceid=502, filesize=11.9 K 2023-03-26 11:54:52,529 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=0 B/0 for 70cba8f2e9bfa903d814db3e7b9776da in 51ms, sequenceid=502, compaction requested=true 2023-03-26 11:54:52,529 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:52,529 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,532 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40614ac2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,535 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,535 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,537 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,537 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,540 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,540 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,540 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,542 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,544 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,548 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,548 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,548 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,548 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:52,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,648 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,664 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,666 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,667 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,668 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,668 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,668 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:52,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3f3ef6c3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,680 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,680 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,688 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,688 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,695 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,695 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:52,713 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:52,713 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:52,713 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:52,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:52,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:52,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:52,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:52,739 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831752737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,740 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831752737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,740 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831752738, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,795 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,798 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,800 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,801 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,801 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,802 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,802 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,802 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:52,802 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,802 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,805 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,805 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,805 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2785a5a2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,809 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,809 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,809 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,810 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,813 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,813 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,813 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,820 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,820 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,820 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,820 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,821 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,822 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,828 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,828 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,828 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,828 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:52,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,828 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,829 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,829 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,829 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831752840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831752841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,843 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:52,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831752841, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,929 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:52,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:52,931 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:52,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:52,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:52,932 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:52,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:52,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:52,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,934 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,934 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,934 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:52,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,934 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,935 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:52,935 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:52,935 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:52,935 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:52,936 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,936 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:52,936 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:52,936 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:52,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:52,937 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,937 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,937 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,938 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:52,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:52,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@acb8b6a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:52,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:52,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:52,940 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,940 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:52,943 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:52,943 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,943 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,943 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:52,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:52,945 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:52,945 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:52,945 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:52,945 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:52,945 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:52,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:52,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,949 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:52,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:52,951 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:52,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:52,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:52,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:52,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:52,951 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:52,952 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:52,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:52,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:52,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:53,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831753044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,044 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831753044, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831753045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,052 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:53,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:53,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:53,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:53,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:53,056 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:53,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:53,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:53,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:53,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,058 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,058 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:53,058 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:53,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:53,058 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,059 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:53,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:53,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:53,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:53,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:53,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:53,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:53,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:53,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:53,061 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:53,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:53,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:53,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:53,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4036e7ab[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:53,066 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:53,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,067 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:53,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:53,070 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:53,071 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:53,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:53,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:53,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:53,073 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:53,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:53,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:53,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:53,073 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:53,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:53,075 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,075 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:53,075 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:53,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:53,076 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:53,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:53,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:53,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:53,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:53,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:53,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:53,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:53,079 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:53,079 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:53,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:53,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:53,116 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831753115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831753123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,126 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/dbcce2818f4b46f9a5a1e2a3519e2628 2023-03-26 11:54:53,137 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/8040fb150dc74dd7a07dc34a9ccfd5a9 2023-03-26 11:54:53,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=515 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/c9131b1badc547acbe13118ee9ef4a1e 2023-03-26 11:54:53,149 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/dbcce2818f4b46f9a5a1e2a3519e2628 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/dbcce2818f4b46f9a5a1e2a3519e2628 2023-03-26 11:54:53,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/dbcce2818f4b46f9a5a1e2a3519e2628, entries=150, sequenceid=515, filesize=11.9 K 2023-03-26 11:54:53,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/8040fb150dc74dd7a07dc34a9ccfd5a9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/8040fb150dc74dd7a07dc34a9ccfd5a9 2023-03-26 11:54:53,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/8040fb150dc74dd7a07dc34a9ccfd5a9, entries=150, sequenceid=515, filesize=11.9 K 2023-03-26 11:54:53,156 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/c9131b1badc547acbe13118ee9ef4a1e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c9131b1badc547acbe13118ee9ef4a1e 2023-03-26 11:54:53,159 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c9131b1badc547acbe13118ee9ef4a1e, entries=150, sequenceid=515, filesize=11.9 K 2023-03-26 11:54:53,160 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 70cba8f2e9bfa903d814db3e7b9776da in 447ms, sequenceid=515, compaction requested=true 2023-03-26 11:54:53,160 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:53,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:53,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:53,160 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:53,160 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:53,160 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:53,162 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:53,162 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:53,162 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:53,162 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/074ece6baa104e37bc708a9c8af82d4e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fc2a2adae6b64235b4ac6a5201b61825, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fdffe2c0855948c7b8303085732d3842, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/8040fb150dc74dd7a07dc34a9ccfd5a9] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=48.6 K 2023-03-26 11:54:53,162 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52240 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:53,162 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:53,162 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:53,162 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 074ece6baa104e37bc708a9c8af82d4e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1679831690656 2023-03-26 11:54:53,162 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/829398fe735741e98a74c19eae5a74e6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/89e173fab6b84e418304b94f48ea757c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/500c73f9988243e591e9f2cec2ecd3e5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/dbcce2818f4b46f9a5a1e2a3519e2628] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=51.0 K 2023-03-26 11:54:53,163 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fc2a2adae6b64235b4ac6a5201b61825, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1679831690992 2023-03-26 11:54:53,163 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 829398fe735741e98a74c19eae5a74e6, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1679831690656 2023-03-26 11:54:53,163 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fdffe2c0855948c7b8303085732d3842, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1679831691556 2023-03-26 11:54:53,163 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 89e173fab6b84e418304b94f48ea757c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1679831690992 2023-03-26 11:54:53,164 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8040fb150dc74dd7a07dc34a9ccfd5a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679831692688 2023-03-26 11:54:53,164 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 500c73f9988243e591e9f2cec2ecd3e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1679831691556 2023-03-26 11:54:53,164 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting dbcce2818f4b46f9a5a1e2a3519e2628, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679831692688 2023-03-26 11:54:53,171 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#311 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:53,171 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:53,179 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:53,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:53,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:53,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:53,196 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:53,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:53,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:53,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:53,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,199 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:53,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:53,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:53,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:53,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:53,200 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:53,219 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:53,219 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:53,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:53,219 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:53,219 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:53,220 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:54:53,220 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:53,220 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:53,220 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:53,220 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:53,220 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:53,220 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:53,229 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/340b2a8b87a84762a8b23ee3df24d001 2023-03-26 11:54:53,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:53,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:53,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831753355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831753358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831753358, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,461 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831753459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,463 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831753462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,464 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831753462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,598 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/9c95b929f52042a88737767290473f7d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/9c95b929f52042a88737767290473f7d 2023-03-26 11:54:53,601 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into 9c95b929f52042a88737767290473f7d(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:53,602 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:53,602 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=12, startTime=1679831693160; duration=0sec 2023-03-26 11:54:53,602 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:53,602 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:53,603 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49800 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:53,603 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:53,603 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:53,603 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9079b852613d491da98b42d9784739b4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f575164fc76450e9dd4cb4f94460bcd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/243d7ff953e243ef84c132ac27b97931, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c9131b1badc547acbe13118ee9ef4a1e] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=48.6 K 2023-03-26 11:54:53,604 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9079b852613d491da98b42d9784739b4, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=467, earliestPutTs=1679831690656 2023-03-26 11:54:53,604 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 6f575164fc76450e9dd4cb4f94460bcd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=488, earliestPutTs=1679831690992 2023-03-26 11:54:53,604 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 243d7ff953e243ef84c132ac27b97931, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=502, earliestPutTs=1679831691556 2023-03-26 11:54:53,605 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c9131b1badc547acbe13118ee9ef4a1e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679831692688 2023-03-26 11:54:53,611 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#315 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:53,619 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/ede5a1ec761147b3b47a28f6967bb137 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ede5a1ec761147b3b47a28f6967bb137 2023-03-26 11:54:53,623 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/524776ee47cf466d83df5a238e24f265 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/524776ee47cf466d83df5a238e24f265 2023-03-26 11:54:53,624 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into ede5a1ec761147b3b47a28f6967bb137(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:53,625 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:53,625 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=12, startTime=1679831693160; duration=0sec 2023-03-26 11:54:53,625 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:53,626 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into 524776ee47cf466d83df5a238e24f265(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:53,626 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:53,626 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=12, startTime=1679831693160; duration=0sec 2023-03-26 11:54:53,627 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:53,643 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/c085ee8aa2c74f55a37ee9e6779198c4 2023-03-26 11:54:53,664 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831753663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831753664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,666 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831753665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,968 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831753966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,969 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831753968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:53,971 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:53,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831753969, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,057 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=535 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/38c5926be3814897b55e49ac0aebd60c 2023-03-26 11:54:54,061 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/340b2a8b87a84762a8b23ee3df24d001 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/340b2a8b87a84762a8b23ee3df24d001 2023-03-26 11:54:54,064 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/340b2a8b87a84762a8b23ee3df24d001, entries=150, sequenceid=535, filesize=11.9 K 2023-03-26 11:54:54,065 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/c085ee8aa2c74f55a37ee9e6779198c4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c085ee8aa2c74f55a37ee9e6779198c4 2023-03-26 11:54:54,069 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c085ee8aa2c74f55a37ee9e6779198c4, entries=150, sequenceid=535, filesize=11.9 K 2023-03-26 11:54:54,069 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/38c5926be3814897b55e49ac0aebd60c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/38c5926be3814897b55e49ac0aebd60c 2023-03-26 11:54:54,073 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/38c5926be3814897b55e49ac0aebd60c, entries=150, sequenceid=535, filesize=11.9 K 2023-03-26 11:54:54,074 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 70cba8f2e9bfa903d814db3e7b9776da in 855ms, sequenceid=535, compaction requested=false 2023-03-26 11:54:54,074 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:54,074 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:54,074 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,076 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:54,084 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:54,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:54,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:54,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8769980[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:54,086 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:54,086 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,088 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:54,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:54,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:54,089 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,089 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:54,090 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:54,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:54,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:54,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:54,093 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:54,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:54,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:54,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:54,093 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:54,096 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,096 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:54,096 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:54,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:54,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:54,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:54,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:54,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:54,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:54,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:54,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:54,106 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:54,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:54,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:54,106 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:54,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:54,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:54,106 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:54,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:54,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,107 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:54,107 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:54,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:54,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:54,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:54,207 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:54,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:54,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:54,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:54,210 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:54,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:54,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:54,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:54,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:54,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:54,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:54,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:54,213 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,213 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:54,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:54,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:54,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:54,214 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:54,214 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:54,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:54,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:54,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:54,214 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:54:54,215 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:54,215 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:54,215 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:54,215 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:54,215 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:54,215 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:54,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:54,472 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:54,515 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831754513, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831754514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831754515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,618 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831754616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831754617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,620 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831754618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,622 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/8a04988f802d4a10a03c7ef56d1b3b80 2023-03-26 11:54:54,631 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/de1082740f1f486096e11e8157743b06 2023-03-26 11:54:54,821 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831754820, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,823 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831754822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:54,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:54,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831754822, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,041 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=551 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/c63c519375054dce93a81276fcdefcc3 2023-03-26 11:54:55,045 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/8a04988f802d4a10a03c7ef56d1b3b80 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8a04988f802d4a10a03c7ef56d1b3b80 2023-03-26 11:54:55,050 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8a04988f802d4a10a03c7ef56d1b3b80, entries=150, sequenceid=551, filesize=11.9 K 2023-03-26 11:54:55,051 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/de1082740f1f486096e11e8157743b06 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/de1082740f1f486096e11e8157743b06 2023-03-26 11:54:55,054 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/de1082740f1f486096e11e8157743b06, entries=150, sequenceid=551, filesize=11.9 K 2023-03-26 11:54:55,055 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/c63c519375054dce93a81276fcdefcc3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c63c519375054dce93a81276fcdefcc3 2023-03-26 11:54:55,059 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c63c519375054dce93a81276fcdefcc3, entries=150, sequenceid=551, filesize=11.9 K 2023-03-26 11:54:55,060 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 70cba8f2e9bfa903d814db3e7b9776da in 846ms, sequenceid=551, compaction requested=true 2023-03-26 11:54:55,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:55,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,061 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,062 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,065 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2548599[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,065 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,065 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,066 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,069 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,072 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,072 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,073 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,073 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,073 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,075 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,077 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,080 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,080 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,080 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:55,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-26 11:54:55,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:55,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:55,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:55,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:55,124 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:55,124 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:55,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831755133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37496 deadline: 1679831755135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,143 DEBUG [Thread-1654] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4145 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:54:55,144 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831755136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,146 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/ee9a71fd593249de915d546bfc2dcb9d 2023-03-26 11:54:55,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831755141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831755141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,180 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,183 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,186 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,187 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,187 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,188 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,188 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,188 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,188 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,190 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,190 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5492de62[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,193 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,193 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,193 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,195 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,197 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,198 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,201 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,202 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,206 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,206 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,206 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,206 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,245 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831755242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,248 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831755246, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831755247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831755247, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,306 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,310 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23ee70e4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,319 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,320 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,320 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,322 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,322 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,322 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,324 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,324 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,324 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,325 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,327 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,327 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,328 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,328 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,328 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,328 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,428 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,430 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,431 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,431 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,431 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,431 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,434 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,434 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,434 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,435 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,435 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,436 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,436 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,436 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,436 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,436 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,436 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,436 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,437 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,441 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe84780[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,444 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,448 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831755446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,450 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831755450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831755450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831755450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,453 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,453 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,453 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,453 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,453 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,455 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,459 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,459 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:55,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,460 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,460 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,560 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,563 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,564 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,566 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,567 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,567 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/2f341e36b1794c88b0dfae9bd1778411 2023-03-26 11:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,567 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,567 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,568 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,568 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,570 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,572 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39b671a9[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,575 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,575 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,576 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,579 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,579 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,581 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,581 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,583 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,584 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,585 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:55,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,587 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,588 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,588 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,688 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,691 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,693 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,693 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,694 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,694 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,695 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,695 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,695 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,695 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,697 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,699 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b95d933[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,699 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,699 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,700 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,701 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,701 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,702 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,702 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,702 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,702 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,705 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,705 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,705 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,705 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,705 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,705 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,707 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,707 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,707 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,709 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,712 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,712 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,712 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,712 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,712 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,712 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,712 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,712 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,752 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831755751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,755 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831755753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,757 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831755755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:55,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831755755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,812 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,815 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,816 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,816 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,817 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,817 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,818 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,818 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,818 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,818 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,819 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,819 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,819 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,819 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,819 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,819 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,819 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,820 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,821 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,821 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,821 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,821 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b2ceff5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,827 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,827 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,827 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,827 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,829 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,829 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,829 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,830 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,832 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,832 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,832 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,833 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,834 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,834 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,834 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,835 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,837 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,837 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,838 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,838 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,838 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:55,838 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,838 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,838 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,838 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,938 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:55,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:55,940 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:55,941 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:55,941 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:55,941 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,941 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:55,941 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:55,941 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,941 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,943 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,943 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,944 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:55,944 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:55,944 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:55,944 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:55,945 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,945 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:55,945 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:55,945 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:55,945 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:55,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,956 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:55,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:55,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76a56957[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:55,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:55,958 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,959 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:55,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:55,959 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,960 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:55,960 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:55,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:55,963 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:55,963 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:55,963 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:55,963 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:55,963 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:55,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,965 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:55,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:55,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:55,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:55,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:55,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:55,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:55,971 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:55,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:55,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:55,972 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:55,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:55,972 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,972 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:55,972 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:55,972 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:55,972 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:55,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:55,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:55,982 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=572 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/d8b27e3f281644e090b9ae1521252bf1 2023-03-26 11:54:55,987 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/ee9a71fd593249de915d546bfc2dcb9d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ee9a71fd593249de915d546bfc2dcb9d 2023-03-26 11:54:55,991 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ee9a71fd593249de915d546bfc2dcb9d, entries=150, sequenceid=572, filesize=11.9 K 2023-03-26 11:54:55,992 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/2f341e36b1794c88b0dfae9bd1778411 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2f341e36b1794c88b0dfae9bd1778411 2023-03-26 11:54:55,996 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2f341e36b1794c88b0dfae9bd1778411, entries=150, sequenceid=572, filesize=11.9 K 2023-03-26 11:54:55,997 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/d8b27e3f281644e090b9ae1521252bf1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d8b27e3f281644e090b9ae1521252bf1 2023-03-26 11:54:56,001 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d8b27e3f281644e090b9ae1521252bf1, entries=150, sequenceid=572, filesize=11.9 K 2023-03-26 11:54:56,002 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=60.38 KB/61830 for 70cba8f2e9bfa903d814db3e7b9776da in 878ms, sequenceid=572, compaction requested=true 2023-03-26 11:54:56,002 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:56,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:56,002 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:56,002 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:56,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:56,002 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:56,004 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:56,004 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/A is initiating minor compaction (all files) 2023-03-26 11:54:56,004 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/A in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:56,004 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/9c95b929f52042a88737767290473f7d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/340b2a8b87a84762a8b23ee3df24d001, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8a04988f802d4a10a03c7ef56d1b3b80, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ee9a71fd593249de915d546bfc2dcb9d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=48.8 K 2023-03-26 11:54:56,004 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:56,005 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/B is initiating minor compaction (all files) 2023-03-26 11:54:56,005 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/B in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:56,005 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/524776ee47cf466d83df5a238e24f265, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c085ee8aa2c74f55a37ee9e6779198c4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/de1082740f1f486096e11e8157743b06, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2f341e36b1794c88b0dfae9bd1778411] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=48.8 K 2023-03-26 11:54:56,005 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9c95b929f52042a88737767290473f7d, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679831692688 2023-03-26 11:54:56,005 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 524776ee47cf466d83df5a238e24f265, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679831692688 2023-03-26 11:54:56,006 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 340b2a8b87a84762a8b23ee3df24d001, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1679831692732 2023-03-26 11:54:56,006 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c085ee8aa2c74f55a37ee9e6779198c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1679831692732 2023-03-26 11:54:56,006 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8a04988f802d4a10a03c7ef56d1b3b80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1679831693350 2023-03-26 11:54:56,007 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting de1082740f1f486096e11e8157743b06, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1679831693350 2023-03-26 11:54:56,007 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2f341e36b1794c88b0dfae9bd1778411, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1679831694509 2023-03-26 11:54:56,007 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ee9a71fd593249de915d546bfc2dcb9d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1679831694509 2023-03-26 11:54:56,020 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#A#compaction#323 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:56,022 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#B#compaction#324 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:56,054 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/fd0ad3016b114456b1cbd5dfc70ccb1a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/fd0ad3016b114456b1cbd5dfc70ccb1a 2023-03-26 11:54:56,061 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/a97374343ae043e2b9154751a4c72e78 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a97374343ae043e2b9154751a4c72e78 2023-03-26 11:54:56,065 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/B of 70cba8f2e9bfa903d814db3e7b9776da into a97374343ae043e2b9154751a4c72e78(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:56,066 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:56,066 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/B, priority=12, startTime=1679831696002; duration=0sec 2023-03-26 11:54:56,066 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:56,066 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:54:56,067 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49936 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:54:56,067 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 70cba8f2e9bfa903d814db3e7b9776da/C is initiating minor compaction (all files) 2023-03-26 11:54:56,067 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 70cba8f2e9bfa903d814db3e7b9776da/C in TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:56,067 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ede5a1ec761147b3b47a28f6967bb137, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/38c5926be3814897b55e49ac0aebd60c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c63c519375054dce93a81276fcdefcc3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d8b27e3f281644e090b9ae1521252bf1] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp, totalSize=48.8 K 2023-03-26 11:54:56,068 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ede5a1ec761147b3b47a28f6967bb137, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=515, earliestPutTs=1679831692688 2023-03-26 11:54:56,072 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:56,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:56,074 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 38c5926be3814897b55e49ac0aebd60c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=535, earliestPutTs=1679831692732 2023-03-26 11:54:56,075 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c63c519375054dce93a81276fcdefcc3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=551, earliestPutTs=1679831693350 2023-03-26 11:54:56,076 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/A of 70cba8f2e9bfa903d814db3e7b9776da into fd0ad3016b114456b1cbd5dfc70ccb1a(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:56,076 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:56,076 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/A, priority=12, startTime=1679831696002; duration=0sec 2023-03-26 11:54:56,076 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:56,076 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d8b27e3f281644e090b9ae1521252bf1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=572, earliestPutTs=1679831694509 2023-03-26 11:54:56,080 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:54:56,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:54:56,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:54:56,088 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:54:56,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:56,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:54:56,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:54:56,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:56,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:56,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:56,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:56,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:56,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:54:56,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:56,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:54:56,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:56,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:54:56,103 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 70cba8f2e9bfa903d814db3e7b9776da#C#compaction#325 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:54:56,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:54:56,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:54:56,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:54:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:54:56,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:54:56,128 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:56,128 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. started... 2023-03-26 11:54:56,128 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:56,130 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:56,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:56,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. as already flushing 2023-03-26 11:54:56,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831756274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831756275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831756275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,278 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831756276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831756377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,381 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831756379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831756379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831756379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,564 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/7b20b4a899e64e4b8a0d8c2f3db2072d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/7b20b4a899e64e4b8a0d8c2f3db2072d 2023-03-26 11:54:56,568 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 70cba8f2e9bfa903d814db3e7b9776da/C of 70cba8f2e9bfa903d814db3e7b9776da into 7b20b4a899e64e4b8a0d8c2f3db2072d(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:54:56,568 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:56,568 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da., storeName=70cba8f2e9bfa903d814db3e7b9776da/C, priority=12, startTime=1679831696002; duration=0sec 2023-03-26 11:54:56,568 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:56,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831756581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831756583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831756583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831756584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,595 DEBUG [Thread-1665] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f8ff99f to 127.0.0.1:53955 2023-03-26 11:54:56,595 DEBUG [Thread-1661] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e65578e to 127.0.0.1:53955 2023-03-26 11:54:56,595 DEBUG [Thread-1665] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:56,607 DEBUG [Thread-1661] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:56,608 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/21aa944ef9674e5aaf21fb8c4db8c4e8 2023-03-26 11:54:56,609 DEBUG [Thread-1657] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x087b3095 to 127.0.0.1:53955 2023-03-26 11:54:56,609 DEBUG [Thread-1657] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:56,609 DEBUG [Thread-1659] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12115052 to 127.0.0.1:53955 2023-03-26 11:54:56,609 DEBUG [Thread-1659] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:56,609 DEBUG [Thread-1663] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6fcbaba8 to 127.0.0.1:53955 2023-03-26 11:54:56,610 DEBUG [Thread-1663] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:56,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831756885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,885 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831756885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831756886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:56,886 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:56,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831756886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,017 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/d95fbf25c6ca44d695fd49298ee9a544 2023-03-26 11:54:57,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:57,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37516 deadline: 1679831757387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37480 deadline: 1679831757387, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,389 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:57,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37508 deadline: 1679831757389, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,390 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:54:57,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:37552 deadline: 1679831757390, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,426 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=586 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/80f2f8ca99b24e10a8b2469db28466d2 2023-03-26 11:54:57,429 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/21aa944ef9674e5aaf21fb8c4db8c4e8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/21aa944ef9674e5aaf21fb8c4db8c4e8 2023-03-26 11:54:57,432 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/21aa944ef9674e5aaf21fb8c4db8c4e8, entries=150, sequenceid=586, filesize=11.9 K 2023-03-26 11:54:57,433 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/d95fbf25c6ca44d695fd49298ee9a544 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d95fbf25c6ca44d695fd49298ee9a544 2023-03-26 11:54:57,436 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d95fbf25c6ca44d695fd49298ee9a544, entries=150, sequenceid=586, filesize=11.9 K 2023-03-26 11:54:57,436 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/80f2f8ca99b24e10a8b2469db28466d2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/80f2f8ca99b24e10a8b2469db28466d2 2023-03-26 11:54:57,439 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/80f2f8ca99b24e10a8b2469db28466d2, entries=150, sequenceid=586, filesize=11.9 K 2023-03-26 11:54:57,440 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 70cba8f2e9bfa903d814db3e7b9776da in 1312ms, sequenceid=586, compaction requested=false 2023-03-26 11:54:57,440 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:57,440 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:54:57,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,441 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:57,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:57,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c147513[Count = 0] remaining members to acquire global barrier 2023-03-26 11:54:57,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:54:57,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,444 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:54:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:54:57,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:54:57,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:54:57,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:54:57,450 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,450 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:54:57,452 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:54:57,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:54:57,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:54:57,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:54:57,452 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:54:57,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:54:57,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:57,454 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:54:57,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:54:57,455 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:57,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:54:57,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:54:57,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:57,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:54:57,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:54:57,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:54:57,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:54:57,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:54:57,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:54:57,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:54:57,458 DEBUG [Thread-1656] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:54:57,458 DEBUG [Thread-1656] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:54:57,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:54:57,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:57,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:54:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:54:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:54:57,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:54:57,558 DEBUG [Thread-1656] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:54:57,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:54:58,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:58,391 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-03-26 11:54:58,391 DEBUG [Thread-1646] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x00ca898c to 127.0.0.1:53955 2023-03-26 11:54:58,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:58,391 DEBUG [Thread-1646] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:58,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:58,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:58,391 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:58,391 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:58,392 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:54:58,393 DEBUG [Thread-1648] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6585a70a to 127.0.0.1:53955 2023-03-26 11:54:58,393 DEBUG [Thread-1648] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:58,395 DEBUG [Thread-1650] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7480f9da to 127.0.0.1:53955 2023-03-26 11:54:58,395 DEBUG [Thread-1650] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:58,396 DEBUG [Thread-1652] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a24d617 to 127.0.0.1:53955 2023-03-26 11:54:58,396 DEBUG [Thread-1652] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:58,799 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/118d124844c744498ca011ad3397a708 2023-03-26 11:54:59,178 DEBUG [Thread-1654] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x104dbf4e to 127.0.0.1:53955 2023-03-26 11:54:59,178 DEBUG [Thread-1654] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:59,178 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 77 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 114 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 81 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 5320 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 5070 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 5080 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 5341 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 5085 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 11:54:59,179 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:54:59,179 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x262f4570 to 127.0.0.1:53955 2023-03-26 11:54:59,179 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:54:59,180 INFO [Listener at localhost/46409] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 11:54:59,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 11:54:59,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 11:54:59,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 11:54:59,183 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831699183"}]},"ts":"1679831699183"} 2023-03-26 11:54:59,184 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 11:54:59,186 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 11:54:59,186 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, UNASSIGN}] 2023-03-26 11:54:59,187 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, UNASSIGN 2023-03-26 11:54:59,188 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=70cba8f2e9bfa903d814db3e7b9776da, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:54:59,188 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831699188"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831699188"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831699188"}]},"ts":"1679831699188"} 2023-03-26 11:54:59,189 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure 70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:54:59,207 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/c2bddfee329c46ae9506f9a57d61fdb2 2023-03-26 11:54:59,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 11:54:59,340 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:54:59,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 70cba8f2e9bfa903d814db3e7b9776da, disabling compactions & flushes 2023-03-26 11:54:59,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:59,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 11:54:59,615 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=608 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/93adb04779c34e6d897ed048404049b2 2023-03-26 11:54:59,619 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/118d124844c744498ca011ad3397a708 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/118d124844c744498ca011ad3397a708 2023-03-26 11:54:59,621 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/118d124844c744498ca011ad3397a708, entries=150, sequenceid=608, filesize=11.9 K 2023-03-26 11:54:59,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/c2bddfee329c46ae9506f9a57d61fdb2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c2bddfee329c46ae9506f9a57d61fdb2 2023-03-26 11:54:59,625 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c2bddfee329c46ae9506f9a57d61fdb2, entries=150, sequenceid=608, filesize=11.9 K 2023-03-26 11:54:59,625 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/93adb04779c34e6d897ed048404049b2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/93adb04779c34e6d897ed048404049b2 2023-03-26 11:54:59,628 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/93adb04779c34e6d897ed048404049b2, entries=150, sequenceid=608, filesize=11.9 K 2023-03-26 11:54:59,629 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=26.84 KB/27480 for 70cba8f2e9bfa903d814db3e7b9776da in 1238ms, sequenceid=608, compaction requested=true 2023-03-26 11:54:59,629 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:54:59,629 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:59,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:59,629 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:59,629 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. because compaction request was cancelled 2023-03-26 11:54:59,629 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. because compaction request was cancelled 2023-03-26 11:54:59,629 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. after waiting 0 ms 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:54:59,629 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:54:59,630 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 70cba8f2e9bfa903d814db3e7b9776da 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 11:54:59,630 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. because compaction request was cancelled 2023-03-26 11:54:59,630 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=A 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=B 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 70cba8f2e9bfa903d814db3e7b9776da, store=C 2023-03-26 11:54:59,630 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:54:59,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 11:55:00,036 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/a05d33961dd5441688fb96ffbeb37389 2023-03-26 11:55:00,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 11:55:00,445 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/d761073cc67143d08843fec0d3117965 2023-03-26 11:55:00,854 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=615 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/caf58f7f94c84481b679e409f3ae5489 2023-03-26 11:55:00,857 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/A/a05d33961dd5441688fb96ffbeb37389 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/a05d33961dd5441688fb96ffbeb37389 2023-03-26 11:55:00,860 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/a05d33961dd5441688fb96ffbeb37389, entries=150, sequenceid=615, filesize=11.9 K 2023-03-26 11:55:00,860 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/B/d761073cc67143d08843fec0d3117965 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d761073cc67143d08843fec0d3117965 2023-03-26 11:55:00,863 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d761073cc67143d08843fec0d3117965, entries=150, sequenceid=615, filesize=11.9 K 2023-03-26 11:55:00,864 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/.tmp/C/caf58f7f94c84481b679e409f3ae5489 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/caf58f7f94c84481b679e409f3ae5489 2023-03-26 11:55:00,866 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/caf58f7f94c84481b679e409f3ae5489, entries=150, sequenceid=615, filesize=11.9 K 2023-03-26 11:55:00,867 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 70cba8f2e9bfa903d814db3e7b9776da in 1237ms, sequenceid=615, compaction requested=true 2023-03-26 11:55:00,890 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/244ebe9a07d941708af41432027db5b2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3cc26d0b88494e9a853f471b8e3da025, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/33c268c4c78144e3ac26ae18f66e08cc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0949f6fe7aaf4698847749df67b99b9b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/01a88710a9fc461c89d7668b51698433, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/03fedf569c8a4bfe9268afb4c0e6fdf2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/448cd6ae8ee644c3965153976ab9a4e7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1ef21589146b499cb9992b215b90ef02, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0b7f7a3db02142569862116c43f643e3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/68085721b6a145caa3e97e589566cfd6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/4484a252be9f4aa880f30f6f091790a5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ce2e7de6b7584882b111328963c543ce, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1be584f1b6984fd5ae36bb9fedf8c00e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f3ebabdf58ba411ba54dfc56162a23bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/27a7dcec4272422f9cd655c256055d08, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/341ab11293c34fd99d4505cd036007dc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8ba2318e579248b6829175e2a2a429e9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f0d8e9c78eb24186b1e43bff02dd193a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/61efb34b34b544deb86d0f79e6c04017, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3dd3498d26ab4555af426634179b7fde, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1292ddf0e52c42c9ac199daff9af33f1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/da0034299ce842a085e0d9f47741cbe0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/288ddb15be994538979e5d51cba11c45, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c3b8b7691ada4fdfb9c4e2f7841a0386, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/043dc9c1f3584ae8a0f3a11f21848d64, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c011ea29433e4ed5a8b76db9258fc383, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c38cb8526c014bbf93a90e3650933a5c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/48ba7fcb10fb405aa5128888803e640d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/2f8ed318aab74dffa67d0e2a1910f755, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/7bc26f2e30b14b4caf8335f48c8f3bcd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/829398fe735741e98a74c19eae5a74e6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/63e1c360921644aaa9395380f221d9be, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/89e173fab6b84e418304b94f48ea757c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/500c73f9988243e591e9f2cec2ecd3e5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/9c95b929f52042a88737767290473f7d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/dbcce2818f4b46f9a5a1e2a3519e2628, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/340b2a8b87a84762a8b23ee3df24d001, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8a04988f802d4a10a03c7ef56d1b3b80, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ee9a71fd593249de915d546bfc2dcb9d] to archive 2023-03-26 11:55:00,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:00,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/244ebe9a07d941708af41432027db5b2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/244ebe9a07d941708af41432027db5b2 2023-03-26 11:55:00,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3cc26d0b88494e9a853f471b8e3da025 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3cc26d0b88494e9a853f471b8e3da025 2023-03-26 11:55:00,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/33c268c4c78144e3ac26ae18f66e08cc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/33c268c4c78144e3ac26ae18f66e08cc 2023-03-26 11:55:00,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0949f6fe7aaf4698847749df67b99b9b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0949f6fe7aaf4698847749df67b99b9b 2023-03-26 11:55:00,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/01a88710a9fc461c89d7668b51698433 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/01a88710a9fc461c89d7668b51698433 2023-03-26 11:55:00,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/03fedf569c8a4bfe9268afb4c0e6fdf2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/03fedf569c8a4bfe9268afb4c0e6fdf2 2023-03-26 11:55:00,898 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/448cd6ae8ee644c3965153976ab9a4e7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/448cd6ae8ee644c3965153976ab9a4e7 2023-03-26 11:55:00,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1ef21589146b499cb9992b215b90ef02 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1ef21589146b499cb9992b215b90ef02 2023-03-26 11:55:00,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0b7f7a3db02142569862116c43f643e3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/0b7f7a3db02142569862116c43f643e3 2023-03-26 11:55:00,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/68085721b6a145caa3e97e589566cfd6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/68085721b6a145caa3e97e589566cfd6 2023-03-26 11:55:00,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/4484a252be9f4aa880f30f6f091790a5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/4484a252be9f4aa880f30f6f091790a5 2023-03-26 11:55:00,903 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ce2e7de6b7584882b111328963c543ce to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ce2e7de6b7584882b111328963c543ce 2023-03-26 11:55:00,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1be584f1b6984fd5ae36bb9fedf8c00e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1be584f1b6984fd5ae36bb9fedf8c00e 2023-03-26 11:55:00,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f3ebabdf58ba411ba54dfc56162a23bf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f3ebabdf58ba411ba54dfc56162a23bf 2023-03-26 11:55:00,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/27a7dcec4272422f9cd655c256055d08 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/27a7dcec4272422f9cd655c256055d08 2023-03-26 11:55:00,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/341ab11293c34fd99d4505cd036007dc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/341ab11293c34fd99d4505cd036007dc 2023-03-26 11:55:00,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8ba2318e579248b6829175e2a2a429e9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8ba2318e579248b6829175e2a2a429e9 2023-03-26 11:55:00,908 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f0d8e9c78eb24186b1e43bff02dd193a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/f0d8e9c78eb24186b1e43bff02dd193a 2023-03-26 11:55:00,909 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/61efb34b34b544deb86d0f79e6c04017 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/61efb34b34b544deb86d0f79e6c04017 2023-03-26 11:55:00,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3dd3498d26ab4555af426634179b7fde to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/3dd3498d26ab4555af426634179b7fde 2023-03-26 11:55:00,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1292ddf0e52c42c9ac199daff9af33f1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/1292ddf0e52c42c9ac199daff9af33f1 2023-03-26 11:55:00,911 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/da0034299ce842a085e0d9f47741cbe0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/da0034299ce842a085e0d9f47741cbe0 2023-03-26 11:55:00,913 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/288ddb15be994538979e5d51cba11c45 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/288ddb15be994538979e5d51cba11c45 2023-03-26 11:55:00,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c3b8b7691ada4fdfb9c4e2f7841a0386 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c3b8b7691ada4fdfb9c4e2f7841a0386 2023-03-26 11:55:00,915 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/043dc9c1f3584ae8a0f3a11f21848d64 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/043dc9c1f3584ae8a0f3a11f21848d64 2023-03-26 11:55:00,916 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c011ea29433e4ed5a8b76db9258fc383 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c011ea29433e4ed5a8b76db9258fc383 2023-03-26 11:55:00,917 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c38cb8526c014bbf93a90e3650933a5c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/c38cb8526c014bbf93a90e3650933a5c 2023-03-26 11:55:00,918 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/48ba7fcb10fb405aa5128888803e640d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/48ba7fcb10fb405aa5128888803e640d 2023-03-26 11:55:00,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/2f8ed318aab74dffa67d0e2a1910f755 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/2f8ed318aab74dffa67d0e2a1910f755 2023-03-26 11:55:00,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/7bc26f2e30b14b4caf8335f48c8f3bcd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/7bc26f2e30b14b4caf8335f48c8f3bcd 2023-03-26 11:55:00,920 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/829398fe735741e98a74c19eae5a74e6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/829398fe735741e98a74c19eae5a74e6 2023-03-26 11:55:00,921 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/63e1c360921644aaa9395380f221d9be to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/63e1c360921644aaa9395380f221d9be 2023-03-26 11:55:00,922 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/89e173fab6b84e418304b94f48ea757c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/89e173fab6b84e418304b94f48ea757c 2023-03-26 11:55:00,923 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/500c73f9988243e591e9f2cec2ecd3e5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/500c73f9988243e591e9f2cec2ecd3e5 2023-03-26 11:55:00,924 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/9c95b929f52042a88737767290473f7d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/9c95b929f52042a88737767290473f7d 2023-03-26 11:55:00,925 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/dbcce2818f4b46f9a5a1e2a3519e2628 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/dbcce2818f4b46f9a5a1e2a3519e2628 2023-03-26 11:55:00,926 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/340b2a8b87a84762a8b23ee3df24d001 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/340b2a8b87a84762a8b23ee3df24d001 2023-03-26 11:55:00,927 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8a04988f802d4a10a03c7ef56d1b3b80 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/8a04988f802d4a10a03c7ef56d1b3b80 2023-03-26 11:55:00,928 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ee9a71fd593249de915d546bfc2dcb9d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/ee9a71fd593249de915d546bfc2dcb9d 2023-03-26 11:55:00,931 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/00677482611a48d481048c4dc0e5ad20, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/177686278de5491c9e0f400ad445d2f8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a51951ea73324db6bf913bda98f97093, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/05d454f8ec9d4ab9866a06325a87b748, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e559c7a73b6492cabca62f1dde23ab0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9dc7ec1342eb43ce9ad71801e70fbec5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/ccb70db1c56f491d8f2b2ec38b1f433b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f453c9a7905b4e3887465034587ff01e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/50e3df4f9bdc41bcbc43855d30e297de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/92dfa2332e464574aef9eaf148b9aa37, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/3f24ecd1b8d94e6ba5dac7b8304e584a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/db188b8e12304063b0ae03ef91e12f5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2ba6d7f8cbb94bd4a44dace128b69edf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/080ceffcceb94070aa42177a07053a79, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c651f78c0ed44299aff0d1c66fce893e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/0816f12a438941abb13964cd81cfa352, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/30dd23adccdd4bb78acaaf7a7ab846de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f1e2ee6e13b642b3abd60306db5d8e85, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/35d2e192ceb54ce8a63a4c46693e60e1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/be36d797b6bd46c29d856e82c5bc16b7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/726270667f19417cae80d145ee77ba2e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a23f768994af4f08be23845b066a08bb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a0029b4ef86443f4a0e56f5ce3c3deda, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d45ff69737974986b6072a005082d8c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/aea125ad245f47ddac799c61a6b68eb1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/bd2bf18760a2434da98c1cb6082a8eca, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/80154b11b1314b768085dc3ff453d607, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/6208354f094d4807a91596f4760c0f8a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/1ed890fbecec4f50837b8d5f5973fc90, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e03e8226e5e431fbda59c630e978ca3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/074ece6baa104e37bc708a9c8af82d4e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/7579ba9dd4564eec84047a34e3777f5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fc2a2adae6b64235b4ac6a5201b61825, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fdffe2c0855948c7b8303085732d3842, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/524776ee47cf466d83df5a238e24f265, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/8040fb150dc74dd7a07dc34a9ccfd5a9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c085ee8aa2c74f55a37ee9e6779198c4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/de1082740f1f486096e11e8157743b06, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2f341e36b1794c88b0dfae9bd1778411] to archive 2023-03-26 11:55:00,932 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:00,933 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/00677482611a48d481048c4dc0e5ad20 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/00677482611a48d481048c4dc0e5ad20 2023-03-26 11:55:00,935 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/177686278de5491c9e0f400ad445d2f8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/177686278de5491c9e0f400ad445d2f8 2023-03-26 11:55:00,935 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a51951ea73324db6bf913bda98f97093 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a51951ea73324db6bf913bda98f97093 2023-03-26 11:55:00,936 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/05d454f8ec9d4ab9866a06325a87b748 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/05d454f8ec9d4ab9866a06325a87b748 2023-03-26 11:55:00,937 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e559c7a73b6492cabca62f1dde23ab0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e559c7a73b6492cabca62f1dde23ab0 2023-03-26 11:55:00,938 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9dc7ec1342eb43ce9ad71801e70fbec5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9dc7ec1342eb43ce9ad71801e70fbec5 2023-03-26 11:55:00,939 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/ccb70db1c56f491d8f2b2ec38b1f433b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/ccb70db1c56f491d8f2b2ec38b1f433b 2023-03-26 11:55:00,940 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f453c9a7905b4e3887465034587ff01e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f453c9a7905b4e3887465034587ff01e 2023-03-26 11:55:00,941 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/50e3df4f9bdc41bcbc43855d30e297de to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/50e3df4f9bdc41bcbc43855d30e297de 2023-03-26 11:55:00,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/92dfa2332e464574aef9eaf148b9aa37 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/92dfa2332e464574aef9eaf148b9aa37 2023-03-26 11:55:00,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/3f24ecd1b8d94e6ba5dac7b8304e584a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/3f24ecd1b8d94e6ba5dac7b8304e584a 2023-03-26 11:55:00,943 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/db188b8e12304063b0ae03ef91e12f5f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/db188b8e12304063b0ae03ef91e12f5f 2023-03-26 11:55:00,944 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2ba6d7f8cbb94bd4a44dace128b69edf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2ba6d7f8cbb94bd4a44dace128b69edf 2023-03-26 11:55:00,945 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/080ceffcceb94070aa42177a07053a79 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/080ceffcceb94070aa42177a07053a79 2023-03-26 11:55:00,946 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c651f78c0ed44299aff0d1c66fce893e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c651f78c0ed44299aff0d1c66fce893e 2023-03-26 11:55:00,946 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/0816f12a438941abb13964cd81cfa352 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/0816f12a438941abb13964cd81cfa352 2023-03-26 11:55:00,947 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/30dd23adccdd4bb78acaaf7a7ab846de to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/30dd23adccdd4bb78acaaf7a7ab846de 2023-03-26 11:55:00,948 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f1e2ee6e13b642b3abd60306db5d8e85 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/f1e2ee6e13b642b3abd60306db5d8e85 2023-03-26 11:55:00,949 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/35d2e192ceb54ce8a63a4c46693e60e1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/35d2e192ceb54ce8a63a4c46693e60e1 2023-03-26 11:55:00,950 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/be36d797b6bd46c29d856e82c5bc16b7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/be36d797b6bd46c29d856e82c5bc16b7 2023-03-26 11:55:00,950 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/726270667f19417cae80d145ee77ba2e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/726270667f19417cae80d145ee77ba2e 2023-03-26 11:55:00,951 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a23f768994af4f08be23845b066a08bb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a23f768994af4f08be23845b066a08bb 2023-03-26 11:55:00,952 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a0029b4ef86443f4a0e56f5ce3c3deda to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a0029b4ef86443f4a0e56f5ce3c3deda 2023-03-26 11:55:00,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d45ff69737974986b6072a005082d8c1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d45ff69737974986b6072a005082d8c1 2023-03-26 11:55:00,953 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/aea125ad245f47ddac799c61a6b68eb1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/aea125ad245f47ddac799c61a6b68eb1 2023-03-26 11:55:00,954 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/bd2bf18760a2434da98c1cb6082a8eca to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/bd2bf18760a2434da98c1cb6082a8eca 2023-03-26 11:55:00,955 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/80154b11b1314b768085dc3ff453d607 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/80154b11b1314b768085dc3ff453d607 2023-03-26 11:55:00,955 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/6208354f094d4807a91596f4760c0f8a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/6208354f094d4807a91596f4760c0f8a 2023-03-26 11:55:00,956 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/1ed890fbecec4f50837b8d5f5973fc90 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/1ed890fbecec4f50837b8d5f5973fc90 2023-03-26 11:55:00,957 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e03e8226e5e431fbda59c630e978ca3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/9e03e8226e5e431fbda59c630e978ca3 2023-03-26 11:55:00,958 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/074ece6baa104e37bc708a9c8af82d4e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/074ece6baa104e37bc708a9c8af82d4e 2023-03-26 11:55:00,958 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/7579ba9dd4564eec84047a34e3777f5f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/7579ba9dd4564eec84047a34e3777f5f 2023-03-26 11:55:00,959 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fc2a2adae6b64235b4ac6a5201b61825 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fc2a2adae6b64235b4ac6a5201b61825 2023-03-26 11:55:00,960 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fdffe2c0855948c7b8303085732d3842 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/fdffe2c0855948c7b8303085732d3842 2023-03-26 11:55:00,961 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/524776ee47cf466d83df5a238e24f265 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/524776ee47cf466d83df5a238e24f265 2023-03-26 11:55:00,962 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/8040fb150dc74dd7a07dc34a9ccfd5a9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/8040fb150dc74dd7a07dc34a9ccfd5a9 2023-03-26 11:55:00,963 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c085ee8aa2c74f55a37ee9e6779198c4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c085ee8aa2c74f55a37ee9e6779198c4 2023-03-26 11:55:00,964 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/de1082740f1f486096e11e8157743b06 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/de1082740f1f486096e11e8157743b06 2023-03-26 11:55:00,965 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2f341e36b1794c88b0dfae9bd1778411 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/2f341e36b1794c88b0dfae9bd1778411 2023-03-26 11:55:00,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f752f71d084548998dde7aed432cd5e6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f6bc5c8580e74a2ab1afc56ace80b1d7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/0c2aa69a511542ea9d47c97dc4326b4f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/886a2f8e682c4455ac4371bc3e4db001, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b59c3cc9efd94df6905233e0fa14cbcf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f3a9a9ad3bc940cba29222b85dcd2fc6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/cc6837e9aef841829f93d230dab71f15, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d545b04a05694d76b0e9650273a6aabf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/245132a9e01540cb835957613b854f8a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ba640aaf185e4e98ac96786ebf166604, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f264318a1384e8691479cba975935b4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/da3e350050624988bdbba54ff90d2d7e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8fce94974b0b47bfbf7f13801a473ed0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8bfcc702fb6e41b48fc29aa5b9cd317b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c7e56d7733014a73be2665e042794516, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/a38e648d99c14b33acc45c183395602c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b1823c6ed85e4633aad39004a863b385, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/084b148255c64138af39d212e008c32d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/493211f4dc494247a275a80cba67b190, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/dc871f23aae74bf89a5c97ac7ae10c43, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ec9bc392c07d4ea1ab03772d05d8ea17, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/99aa04d29bae443195b822cce5864555, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9f8e8b4b4d69492f88c8de5b60f08046, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/896756b1926f40358ed235a22ecfbb0c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/322478c7f58143c4a4130ae7ff1e4207, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/750f2bbcafb349f7b92945b435b8d97b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8327754de4b54561b4770f5267cbce78, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1143f4a0c9f843e4894bfc2868e5b0cd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/4b7c5af9419540a88fa9c122e73efd60, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d6ec087222004c7caedb0a57d06dba3e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9079b852613d491da98b42d9784739b4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1e54ee9f9165495f81d0507e5a02c7aa, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f575164fc76450e9dd4cb4f94460bcd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/243d7ff953e243ef84c132ac27b97931, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ede5a1ec761147b3b47a28f6967bb137, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c9131b1badc547acbe13118ee9ef4a1e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/38c5926be3814897b55e49ac0aebd60c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c63c519375054dce93a81276fcdefcc3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d8b27e3f281644e090b9ae1521252bf1] to archive 2023-03-26 11:55:00,968 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:00,969 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f752f71d084548998dde7aed432cd5e6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f752f71d084548998dde7aed432cd5e6 2023-03-26 11:55:00,970 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f6bc5c8580e74a2ab1afc56ace80b1d7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f6bc5c8580e74a2ab1afc56ace80b1d7 2023-03-26 11:55:00,971 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/0c2aa69a511542ea9d47c97dc4326b4f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/0c2aa69a511542ea9d47c97dc4326b4f 2023-03-26 11:55:00,972 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/886a2f8e682c4455ac4371bc3e4db001 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/886a2f8e682c4455ac4371bc3e4db001 2023-03-26 11:55:00,973 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b59c3cc9efd94df6905233e0fa14cbcf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b59c3cc9efd94df6905233e0fa14cbcf 2023-03-26 11:55:00,974 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f3a9a9ad3bc940cba29222b85dcd2fc6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/f3a9a9ad3bc940cba29222b85dcd2fc6 2023-03-26 11:55:00,975 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/cc6837e9aef841829f93d230dab71f15 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/cc6837e9aef841829f93d230dab71f15 2023-03-26 11:55:00,975 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d545b04a05694d76b0e9650273a6aabf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d545b04a05694d76b0e9650273a6aabf 2023-03-26 11:55:00,976 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/245132a9e01540cb835957613b854f8a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/245132a9e01540cb835957613b854f8a 2023-03-26 11:55:00,977 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ba640aaf185e4e98ac96786ebf166604 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ba640aaf185e4e98ac96786ebf166604 2023-03-26 11:55:00,978 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f264318a1384e8691479cba975935b4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f264318a1384e8691479cba975935b4 2023-03-26 11:55:00,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/da3e350050624988bdbba54ff90d2d7e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/da3e350050624988bdbba54ff90d2d7e 2023-03-26 11:55:00,979 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8fce94974b0b47bfbf7f13801a473ed0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8fce94974b0b47bfbf7f13801a473ed0 2023-03-26 11:55:00,980 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8bfcc702fb6e41b48fc29aa5b9cd317b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8bfcc702fb6e41b48fc29aa5b9cd317b 2023-03-26 11:55:00,981 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c7e56d7733014a73be2665e042794516 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c7e56d7733014a73be2665e042794516 2023-03-26 11:55:00,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/a38e648d99c14b33acc45c183395602c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/a38e648d99c14b33acc45c183395602c 2023-03-26 11:55:00,982 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b1823c6ed85e4633aad39004a863b385 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/b1823c6ed85e4633aad39004a863b385 2023-03-26 11:55:00,983 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/084b148255c64138af39d212e008c32d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/084b148255c64138af39d212e008c32d 2023-03-26 11:55:00,984 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/493211f4dc494247a275a80cba67b190 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/493211f4dc494247a275a80cba67b190 2023-03-26 11:55:00,985 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/dc871f23aae74bf89a5c97ac7ae10c43 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/dc871f23aae74bf89a5c97ac7ae10c43 2023-03-26 11:55:00,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ec9bc392c07d4ea1ab03772d05d8ea17 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ec9bc392c07d4ea1ab03772d05d8ea17 2023-03-26 11:55:00,986 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/99aa04d29bae443195b822cce5864555 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/99aa04d29bae443195b822cce5864555 2023-03-26 11:55:00,987 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9f8e8b4b4d69492f88c8de5b60f08046 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9f8e8b4b4d69492f88c8de5b60f08046 2023-03-26 11:55:00,988 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/896756b1926f40358ed235a22ecfbb0c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/896756b1926f40358ed235a22ecfbb0c 2023-03-26 11:55:00,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/322478c7f58143c4a4130ae7ff1e4207 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/322478c7f58143c4a4130ae7ff1e4207 2023-03-26 11:55:00,989 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/750f2bbcafb349f7b92945b435b8d97b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/750f2bbcafb349f7b92945b435b8d97b 2023-03-26 11:55:00,990 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8327754de4b54561b4770f5267cbce78 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/8327754de4b54561b4770f5267cbce78 2023-03-26 11:55:00,991 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1143f4a0c9f843e4894bfc2868e5b0cd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1143f4a0c9f843e4894bfc2868e5b0cd 2023-03-26 11:55:00,992 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/4b7c5af9419540a88fa9c122e73efd60 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/4b7c5af9419540a88fa9c122e73efd60 2023-03-26 11:55:00,993 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d6ec087222004c7caedb0a57d06dba3e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d6ec087222004c7caedb0a57d06dba3e 2023-03-26 11:55:00,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9079b852613d491da98b42d9784739b4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/9079b852613d491da98b42d9784739b4 2023-03-26 11:55:00,994 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1e54ee9f9165495f81d0507e5a02c7aa to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/1e54ee9f9165495f81d0507e5a02c7aa 2023-03-26 11:55:00,995 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f575164fc76450e9dd4cb4f94460bcd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/6f575164fc76450e9dd4cb4f94460bcd 2023-03-26 11:55:00,996 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/243d7ff953e243ef84c132ac27b97931 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/243d7ff953e243ef84c132ac27b97931 2023-03-26 11:55:00,997 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ede5a1ec761147b3b47a28f6967bb137 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/ede5a1ec761147b3b47a28f6967bb137 2023-03-26 11:55:00,998 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c9131b1badc547acbe13118ee9ef4a1e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c9131b1badc547acbe13118ee9ef4a1e 2023-03-26 11:55:00,999 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/38c5926be3814897b55e49ac0aebd60c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/38c5926be3814897b55e49ac0aebd60c 2023-03-26 11:55:01,000 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c63c519375054dce93a81276fcdefcc3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/c63c519375054dce93a81276fcdefcc3 2023-03-26 11:55:01,001 DEBUG [StoreCloser-TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d8b27e3f281644e090b9ae1521252bf1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/d8b27e3f281644e090b9ae1521252bf1 2023-03-26 11:55:01,004 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/recovered.edits/618.seqid, newMaxSeqId=618, maxSeqId=1 2023-03-26 11:55:01,005 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da. 2023-03-26 11:55:01,005 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 70cba8f2e9bfa903d814db3e7b9776da: 2023-03-26 11:55:01,007 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:55:01,007 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=70cba8f2e9bfa903d814db3e7b9776da, regionState=CLOSED 2023-03-26 11:55:01,007 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831701007"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831701007"}]},"ts":"1679831701007"} 2023-03-26 11:55:01,009 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-03-26 11:55:01,009 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure 70cba8f2e9bfa903d814db3e7b9776da, server=jenkins-hbase4.apache.org,38971,1679831615531 in 1.8190 sec 2023-03-26 11:55:01,010 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-03-26 11:55:01,010 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=70cba8f2e9bfa903d814db3e7b9776da, UNASSIGN in 1.8230 sec 2023-03-26 11:55:01,011 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831701011"}]},"ts":"1679831701011"} 2023-03-26 11:55:01,012 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 11:55:01,016 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 11:55:01,018 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8370 sec 2023-03-26 11:55:01,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-03-26 11:55:01,287 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-03-26 11:55:01,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 11:55:01,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,290 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-26 11:55:01,291 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,295 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:55:01,297 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/recovered.edits] 2023-03-26 11:55:01,299 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/118d124844c744498ca011ad3397a708 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/118d124844c744498ca011ad3397a708 2023-03-26 11:55:01,300 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/21aa944ef9674e5aaf21fb8c4db8c4e8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/21aa944ef9674e5aaf21fb8c4db8c4e8 2023-03-26 11:55:01,301 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/a05d33961dd5441688fb96ffbeb37389 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/a05d33961dd5441688fb96ffbeb37389 2023-03-26 11:55:01,302 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/fd0ad3016b114456b1cbd5dfc70ccb1a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/A/fd0ad3016b114456b1cbd5dfc70ccb1a 2023-03-26 11:55:01,304 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a97374343ae043e2b9154751a4c72e78 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/a97374343ae043e2b9154751a4c72e78 2023-03-26 11:55:01,305 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c2bddfee329c46ae9506f9a57d61fdb2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/c2bddfee329c46ae9506f9a57d61fdb2 2023-03-26 11:55:01,305 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d761073cc67143d08843fec0d3117965 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d761073cc67143d08843fec0d3117965 2023-03-26 11:55:01,306 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d95fbf25c6ca44d695fd49298ee9a544 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/B/d95fbf25c6ca44d695fd49298ee9a544 2023-03-26 11:55:01,308 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/7b20b4a899e64e4b8a0d8c2f3db2072d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/7b20b4a899e64e4b8a0d8c2f3db2072d 2023-03-26 11:55:01,309 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/80f2f8ca99b24e10a8b2469db28466d2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/80f2f8ca99b24e10a8b2469db28466d2 2023-03-26 11:55:01,310 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/93adb04779c34e6d897ed048404049b2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/93adb04779c34e6d897ed048404049b2 2023-03-26 11:55:01,311 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/caf58f7f94c84481b679e409f3ae5489 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/C/caf58f7f94c84481b679e409f3ae5489 2023-03-26 11:55:01,313 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/recovered.edits/618.seqid to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da/recovered.edits/618.seqid 2023-03-26 11:55:01,314 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/70cba8f2e9bfa903d814db3e7b9776da 2023-03-26 11:55:01,314 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:55:01,316 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,317 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 11:55:01,318 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 11:55:01,319 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,319 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 11:55:01,319 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679831701319"}]},"ts":"9223372036854775807"} 2023-03-26 11:55:01,320 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 11:55:01,320 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 70cba8f2e9bfa903d814db3e7b9776da, NAME => 'TestAcidGuarantees,,1679831674428.70cba8f2e9bfa903d814db3e7b9776da.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 11:55:01,320 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 11:55:01,320 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679831701320"}]},"ts":"9223372036854775807"} 2023-03-26 11:55:01,322 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 11:55:01,323 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-03-26 11:55:01,391 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-03-26 11:55:01,392 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-03-26 11:55:01,403 INFO [Listener at localhost/46409] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=228 (was 234), OpenFileDescriptor=569 (was 583), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=366 (was 404), ProcessCount=169 (was 169), AvailableMemoryMB=8831 (was 8905) 2023-03-26 11:55:01,414 INFO [Listener at localhost/46409] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=366, ProcessCount=169, AvailableMemoryMB=8831 2023-03-26 11:55:01,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:55:01,416 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:55:01,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:01,419 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:55:01,419 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-03-26 11:55:01,419 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:55:01,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 11:55:01,420 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:01,421 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 empty. 2023-03-26 11:55:01,421 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:01,421 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:55:01,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 11:55:01,721 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 11:55:01,828 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:55:01,829 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => b57c4d4a48d50ba171b46d52c705a057, NAME => 'TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:55:02,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 11:55:02,235 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:55:02,235 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing b57c4d4a48d50ba171b46d52c705a057, disabling compactions & flushes 2023-03-26 11:55:02,235 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,235 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,235 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. after waiting 0 ms 2023-03-26 11:55:02,235 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,235 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,235 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:02,237 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:55:02,238 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831702238"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831702238"}]},"ts":"1679831702238"} 2023-03-26 11:55:02,239 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:55:02,240 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:55:02,240 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831702240"}]},"ts":"1679831702240"} 2023-03-26 11:55:02,241 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 11:55:02,244 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, ASSIGN}] 2023-03-26 11:55:02,245 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, ASSIGN 2023-03-26 11:55:02,246 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:55:02,397 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:02,397 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831702397"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831702397"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831702397"}]},"ts":"1679831702397"} 2023-03-26 11:55:02,398 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:55:02,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 11:55:02,553 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b57c4d4a48d50ba171b46d52c705a057, NAME => 'TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:55:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:55:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,553 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,554 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,556 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A 2023-03-26 11:55:02,556 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A 2023-03-26 11:55:02,556 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:02,556 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b57c4d4a48d50ba171b46d52c705a057 columnFamilyName A 2023-03-26 11:55:02,556 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(310): Store=b57c4d4a48d50ba171b46d52c705a057/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:02,557 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,558 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B 2023-03-26 11:55:02,558 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B 2023-03-26 11:55:02,558 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:02,558 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b57c4d4a48d50ba171b46d52c705a057 columnFamilyName B 2023-03-26 11:55:02,558 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(310): Store=b57c4d4a48d50ba171b46d52c705a057/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:02,558 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,559 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C 2023-03-26 11:55:02,559 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C 2023-03-26 11:55:02,559 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:02,560 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b57c4d4a48d50ba171b46d52c705a057 columnFamilyName C 2023-03-26 11:55:02,560 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(310): Store=b57c4d4a48d50ba171b46d52c705a057/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:02,560 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,561 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,562 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:55:02,563 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:02,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:55:02,565 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b57c4d4a48d50ba171b46d52c705a057; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66312223, jitterRate=-0.01187087595462799}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:55:02,565 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:02,566 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., pid=37, masterSystemTime=1679831702550 2023-03-26 11:55:02,567 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,567 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:02,567 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:02,567 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831702567"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831702567"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831702567"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831702567"}]},"ts":"1679831702567"} 2023-03-26 11:55:02,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-03-26 11:55:02,569 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 in 170 msec 2023-03-26 11:55:02,570 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-03-26 11:55:02,571 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, ASSIGN in 325 msec 2023-03-26 11:55:02,571 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:55:02,571 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831702571"}]},"ts":"1679831702571"} 2023-03-26 11:55:02,572 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 11:55:02,574 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:55:02,574 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1570 sec 2023-03-26 11:55:03,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-03-26 11:55:03,523 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-03-26 11:55:03,524 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x0bb27fd4 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:03,529 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@49b5bd0d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:03,530 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:03,532 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60094, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:03,533 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:55:03,534 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:49648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:55:03,535 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:55:03,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:55:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:03,945 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 11:55:03,946 DEBUG [PEWorker-4] util.FSTableDescriptors(519): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:55:03,946 INFO [PEWorker-4] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 11:55:03,948 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-26 11:55:03,950 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, REOPEN/MOVE}] 2023-03-26 11:55:03,951 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, REOPEN/MOVE 2023-03-26 11:55:03,951 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:03,951 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831703951"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831703951"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831703951"}]},"ts":"1679831703951"} 2023-03-26 11:55:03,952 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:55:04,104 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b57c4d4a48d50ba171b46d52c705a057, disabling compactions & flushes 2023-03-26 11:55:04,105 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. after waiting 0 ms 2023-03-26 11:55:04,105 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-26 11:55:04,108 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,108 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:04,108 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: b57c4d4a48d50ba171b46d52c705a057 to self. 2023-03-26 11:55:04,110 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,110 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=CLOSED 2023-03-26 11:55:04,110 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831704110"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831704110"}]},"ts":"1679831704110"} 2023-03-26 11:55:04,112 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-03-26 11:55:04,112 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 in 159 msec 2023-03-26 11:55:04,113 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=true 2023-03-26 11:55:04,263 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,263 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831704263"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831704263"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831704263"}]},"ts":"1679831704263"} 2023-03-26 11:55:04,264 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:55:04,419 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,419 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => b57c4d4a48d50ba171b46d52c705a057, NAME => 'TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:55:04,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:55:04,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,420 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,421 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,422 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A 2023-03-26 11:55:04,422 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A 2023-03-26 11:55:04,422 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:04,422 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b57c4d4a48d50ba171b46d52c705a057 columnFamilyName A 2023-03-26 11:55:04,424 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(310): Store=b57c4d4a48d50ba171b46d52c705a057/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:04,424 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,424 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B 2023-03-26 11:55:04,424 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B 2023-03-26 11:55:04,425 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:04,425 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b57c4d4a48d50ba171b46d52c705a057 columnFamilyName B 2023-03-26 11:55:04,425 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(310): Store=b57c4d4a48d50ba171b46d52c705a057/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:04,425 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,426 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C 2023-03-26 11:55:04,426 DEBUG [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C 2023-03-26 11:55:04,426 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:04,426 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region b57c4d4a48d50ba171b46d52c705a057 columnFamilyName C 2023-03-26 11:55:04,427 INFO [StoreOpener-b57c4d4a48d50ba171b46d52c705a057-1] regionserver.HStore(310): Store=b57c4d4a48d50ba171b46d52c705a057/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:04,427 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,427 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,428 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,430 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:55:04,431 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened b57c4d4a48d50ba171b46d52c705a057; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61397192, jitterRate=-0.08511054515838623}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:55:04,432 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:04,432 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., pid=42, masterSystemTime=1679831704416 2023-03-26 11:55:04,433 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,433 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,434 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,434 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831704434"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831704434"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831704434"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831704434"}]},"ts":"1679831704434"} 2023-03-26 11:55:04,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-03-26 11:55:04,436 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 in 171 msec 2023-03-26 11:55:04,437 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-03-26 11:55:04,437 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, REOPEN/MOVE in 486 msec 2023-03-26 11:55:04,438 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-03-26 11:55:04,439 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 489 msec 2023-03-26 11:55:04,440 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 903 msec 2023-03-26 11:55:04,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-03-26 11:55:04,442 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x445919fd to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,446 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c8d927d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,447 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x4d010713 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,450 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@419b69bf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,451 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x69100738 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,455 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c6cac96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,456 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6f603c27 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,459 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a0df34, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,459 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x7e01d5e5 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,462 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10ee5f06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,462 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x21128196 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,466 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4366fcdd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,466 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x5f9e86d9 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,472 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6a35a28d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,472 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x711ebaf2 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,477 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5a1d4246, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,477 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x0a7503d4 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,481 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eb8f142, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,481 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x7eae4191 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:04,484 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5ce79633, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:04,490 DEBUG [hconnection-0x6693b4d2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,490 DEBUG [hconnection-0x1ff768cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,490 DEBUG [hconnection-0x6454c880-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,490 DEBUG [hconnection-0x414e4293-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:04,492 DEBUG [hconnection-0x4b7b0f3c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,493 DEBUG [hconnection-0x33aa2824-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,493 DEBUG [hconnection-0x79927ed3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,493 DEBUG [hconnection-0x23680d45-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,492 DEBUG [hconnection-0x4feac512-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,492 DEBUG [hconnection-0x15165fbc-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:04,492 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60104, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,492 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60120, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,492 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60126, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,495 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60140, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,496 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60154, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,497 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60184, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,498 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60134, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,499 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60168, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,499 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60194, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,499 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:60196, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:04,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:04,501 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:04,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:04,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:04,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:04,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:04,506 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:04,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:04,506 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:04,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:04,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:04,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:04,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:04,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:04,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:04,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:04,509 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:04,510 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 11:55:04,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:04,511 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:04,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:04,532 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032650f07478b80a4b95aa1f4b4bb57ffa54 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032650f07478b80a4b95aa1f4b4bb57ffa54 2023-03-26 11:55:04,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831704517, max timestamp=1679831704551Num uniques 859; 2023-03-26 11:55:04,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831704517, max timestamp=1679831704551Num uniques 859; 2023-03-26 11:55:04,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:04,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831704517, max timestamp=1679831704551Num uniques 859; 2023-03-26 11:55:04,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:04,554 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:04,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831764570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831764571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831764575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831764576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831764578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831764679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831764682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831764682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831764683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831764683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831764882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831764887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831764888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831764890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:04,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831764890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:04,940 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=14, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/8499a7bcd837430aa09146c0c822a7d2 2023-03-26 11:55:05,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831765190, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831765191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,194 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831765192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831765196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831765196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,351 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4 2023-03-26 11:55:05,363 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=14 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/ce158224643b497f8b96732ad86b16a1 2023-03-26 11:55:05,367 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/8499a7bcd837430aa09146c0c822a7d2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8499a7bcd837430aa09146c0c822a7d2 2023-03-26 11:55:05,370 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8499a7bcd837430aa09146c0c822a7d2, entries=200, sequenceid=14, filesize=31.9 K 2023-03-26 11:55:05,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4 2023-03-26 11:55:05,375 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4, entries=150, sequenceid=14, filesize=11.6 K 2023-03-26 11:55:05,375 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/ce158224643b497f8b96732ad86b16a1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/ce158224643b497f8b96732ad86b16a1 2023-03-26 11:55:05,378 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/ce158224643b497f8b96732ad86b16a1, entries=150, sequenceid=14, filesize=11.6 K 2023-03-26 11:55:05,379 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for b57c4d4a48d50ba171b46d52c705a057 in 869ms, sequenceid=14, compaction requested=false 2023-03-26 11:55:05,379 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:05,379 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:05,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:05,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:05,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:05,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:05,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,380 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:05,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:05,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:05,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@299d67e3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:05,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:05,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,383 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:05,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:05,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:05,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:05,385 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:05,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:05,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:05,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:05,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:05,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:05,387 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:05,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:05,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:05,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:05,387 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:05,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:05,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:05,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:05,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:05,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:05,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:05,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:05,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:05,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:05,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:05,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:05,394 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:05,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:05,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:05,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,394 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:05,394 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:05,495 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:05,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:05,496 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:05,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:05,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:05,498 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:05,500 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:05,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:05,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:05,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:05,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:05,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:05,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:05,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:05,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:05,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:05,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:05,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:05,503 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:05,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:05,698 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:05,698 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:05,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831765703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,708 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831765704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831765706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831765707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,712 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831765707, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,812 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831765809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831765812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831765812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:05,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831765813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:05,917 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303262a2b2648d62a4f588cc6b2513df384cb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262a2b2648d62a4f588cc6b2513df384cb 2023-03-26 11:55:05,930 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/bd97644b2b034450b351bf2bcf1bb9f9 2023-03-26 11:55:06,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831766014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,019 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831766016, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,020 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831766018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831766018, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831766320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831766320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,328 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831766321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,329 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831766323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,342 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/1db86fe3044845b39f63e1c4681cdba9 2023-03-26 11:55:06,386 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/22ba5aaf9ccd49c6860967cafeeea388 2023-03-26 11:55:06,390 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/bd97644b2b034450b351bf2bcf1bb9f9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/bd97644b2b034450b351bf2bcf1bb9f9 2023-03-26 11:55:06,393 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/bd97644b2b034450b351bf2bcf1bb9f9, entries=150, sequenceid=51, filesize=25.1 K 2023-03-26 11:55:06,394 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/1db86fe3044845b39f63e1c4681cdba9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1db86fe3044845b39f63e1c4681cdba9 2023-03-26 11:55:06,398 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1db86fe3044845b39f63e1c4681cdba9, entries=150, sequenceid=51, filesize=11.6 K 2023-03-26 11:55:06,399 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/22ba5aaf9ccd49c6860967cafeeea388 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/22ba5aaf9ccd49c6860967cafeeea388 2023-03-26 11:55:06,404 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/22ba5aaf9ccd49c6860967cafeeea388, entries=150, sequenceid=51, filesize=11.6 K 2023-03-26 11:55:06,405 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for b57c4d4a48d50ba171b46d52c705a057 in 902ms, sequenceid=51, compaction requested=false 2023-03-26 11:55:06,405 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:06,405 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:06,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:06,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:06,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:06,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:06,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:06,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:06,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:06,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65026662[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:06,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:06,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:06,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:06,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:06,416 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,416 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:06,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,417 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:06,417 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:06,417 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:06,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:06,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:06,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:06,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:06,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:06,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:06,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:06,424 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:06,424 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:06,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:06,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:06,425 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:06,425 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:06,426 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,426 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:06,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,427 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:06,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:06,427 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:06,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:06,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,429 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:06,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:06,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:06,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,432 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:06,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:06,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:06,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:06,433 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:06,432 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:06,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:06,432 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:06,433 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:06,433 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,433 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,433 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:06,433 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,433 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:06,433 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:06,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:06,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:06,533 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:06,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:06,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:06,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:06,536 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:06,537 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:06,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:06,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:06,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:06,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:06,538 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:06,538 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:06,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:06,539 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,539 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:06,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:06,539 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:06,540 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:06,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:06,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:06,540 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:06,540 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:06,540 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:06,540 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:06,540 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:06,540 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 11:55:06,540 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:06,541 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:06,541 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:06,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:06,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:06,832 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831706788, max timestamp=1679831706818Num uniques 859; 2023-03-26 11:55:06,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:06,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831706788, max timestamp=1679831706818Num uniques 859; 2023-03-26 11:55:06,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:06,833 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831706788, max timestamp=1679831706818Num uniques 859; 2023-03-26 11:55:06,834 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:55:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831766878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831766879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831766880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,892 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831766881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831766881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,982 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032699cef0f9f60345ebab961d8b45d63c83 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032699cef0f9f60345ebab961d8b45d63c83 2023-03-26 11:55:06,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831766988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831766989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:06,997 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:06,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831766989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,001 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831766993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831766994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831767197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831767197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831767200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831767203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831767204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,400 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=61, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/ee94f970e44a4e33bdcbc76d908494e9 2023-03-26 11:55:07,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831767504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831767505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831767506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,511 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831767510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,512 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:07,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831767511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,812 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/b60c6f4bbf604dcc972049afbd220939 2023-03-26 11:55:07,833 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=61 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/98aefd7e9bd847cb9fbc432d7d09b2da 2023-03-26 11:55:07,838 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/ee94f970e44a4e33bdcbc76d908494e9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/ee94f970e44a4e33bdcbc76d908494e9 2023-03-26 11:55:07,841 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/ee94f970e44a4e33bdcbc76d908494e9, entries=150, sequenceid=61, filesize=25.1 K 2023-03-26 11:55:07,842 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/b60c6f4bbf604dcc972049afbd220939 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b60c6f4bbf604dcc972049afbd220939 2023-03-26 11:55:07,845 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b60c6f4bbf604dcc972049afbd220939, entries=150, sequenceid=61, filesize=11.6 K 2023-03-26 11:55:07,847 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/98aefd7e9bd847cb9fbc432d7d09b2da as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/98aefd7e9bd847cb9fbc432d7d09b2da 2023-03-26 11:55:07,850 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/98aefd7e9bd847cb9fbc432d7d09b2da, entries=150, sequenceid=61, filesize=11.6 K 2023-03-26 11:55:07,851 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=133.78 KB/136995 for b57c4d4a48d50ba171b46d52c705a057 in 1309ms, sequenceid=61, compaction requested=true 2023-03-26 11:55:07,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:07,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:07,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:07,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:07,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:07,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:07,853 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:07,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:07,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:07,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38305444[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:07,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:07,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:07,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:07,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:07,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:07,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:07,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:07,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:07,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:07,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:07,863 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:07,863 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:07,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:07,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:07,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:07,864 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:07,865 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,865 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:07,865 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:07,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:07,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:07,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:07,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:07,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:07,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:07,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:07,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:07,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:07,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:07,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:07,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:07,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:07,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:07,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:07,871 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:07,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:07,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:07,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,871 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:07,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:07,871 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,971 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:07,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:07,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:07,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:07,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:07,974 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:07,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:07,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:07,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:07,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,975 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:07,976 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:07,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:07,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:07,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:07,977 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:07,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:07,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:07,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:07,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:07,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:07,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:07,978 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:07,978 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:07,979 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=133.78 KB heapSize=384.07 KB 2023-03-26 11:55:07,979 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:07,979 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:07,979 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:07,979 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:07,979 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:07,979 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:08,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:08,020 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:08,032 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831768022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831768024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,037 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831768025, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831768031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831768033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831768134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831768138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,142 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831768138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831768144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,153 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831768146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831768341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831768343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831768344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,360 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831768355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831768356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,403 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263944f8f099734a4e9fe6c2b176ae0790 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263944f8f099734a4e9fe6c2b176ae0790 2023-03-26 11:55:08,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831768648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,654 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831768648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,655 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831768650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,666 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,666 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831768663, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,669 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:08,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831768664, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:08,808 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/149e57efef264a8eb1f7df4e74881c7a 2023-03-26 11:55:08,819 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/1b826e2732984a25a0db009d4e7910bb 2023-03-26 11:55:09,160 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:09,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831769155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:09,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831769157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:09,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831769158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:09,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831769170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:09,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831769172, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,229 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.33 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/417818068c224b42b927ec7ff4b40c94 2023-03-26 11:55:09,234 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/149e57efef264a8eb1f7df4e74881c7a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/149e57efef264a8eb1f7df4e74881c7a 2023-03-26 11:55:09,237 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/149e57efef264a8eb1f7df4e74881c7a, entries=150, sequenceid=98, filesize=25.1 K 2023-03-26 11:55:09,238 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/1b826e2732984a25a0db009d4e7910bb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1b826e2732984a25a0db009d4e7910bb 2023-03-26 11:55:09,241 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1b826e2732984a25a0db009d4e7910bb, entries=150, sequenceid=98, filesize=11.6 K 2023-03-26 11:55:09,242 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/417818068c224b42b927ec7ff4b40c94 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/417818068c224b42b927ec7ff4b40c94 2023-03-26 11:55:09,245 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/417818068c224b42b927ec7ff4b40c94, entries=150, sequenceid=98, filesize=11.6 K 2023-03-26 11:55:09,245 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~133.78 KB/136995, heapSize ~384.02 KB/393240, currentSize=46.96 KB/48090 for b57c4d4a48d50ba171b46d52c705a057 in 1266ms, sequenceid=98, compaction requested=true 2023-03-26 11:55:09,245 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:09,245 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:09,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:09,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:09,245 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,246 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:09,246 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:09,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,248 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:09,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:09,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:09,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675312bc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:09,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:09,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,251 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:09,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:09,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:09,251 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,251 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:09,253 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:09,253 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:09,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,253 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:09,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:09,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:09,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:09,255 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:09,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:09,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:09,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:09,255 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:09,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:09,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:09,258 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:09,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:09,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:09,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:09,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:09,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:09,262 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:09,263 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:09,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:09,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:09,363 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:09,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:09,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:09,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:09,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:09,366 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:09,366 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:09,366 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:09,366 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:09,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:09,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,368 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:09,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:09,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:09,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:09,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:09,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:09,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:09,369 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:09,369 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:09,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:09,370 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:09,370 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:09,370 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:09,370 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:09,370 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:09,370 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:09,370 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:55:09,370 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:09,370 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:09,370 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:09,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:09,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:09,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:09,784 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e29a0cc6e03f404bb2b6961cb837c9c5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e29a0cc6e03f404bb2b6961cb837c9c5 2023-03-26 11:55:10,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:10,168 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,194 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=108, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/5662b38139074e14819a354060c97405 2023-03-26 11:55:10,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831710197, max timestamp=1679831710205Num uniques 859; 2023-03-26 11:55:10,210 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:10,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831710197, max timestamp=1679831710205Num uniques 859; 2023-03-26 11:55:10,211 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:10,211 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/9d776a7612e44a50a3d8dd8418c747ba 2023-03-26 11:55:10,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831710197, max timestamp=1679831710205Num uniques 859; 2023-03-26 11:55:10,212 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:55:10,220 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=108 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/06356a2cbbae4d34b7224e45f1be0389 2023-03-26 11:55:10,222 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/5662b38139074e14819a354060c97405 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5662b38139074e14819a354060c97405 2023-03-26 11:55:10,225 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5662b38139074e14819a354060c97405, entries=150, sequenceid=108, filesize=25.1 K 2023-03-26 11:55:10,226 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/9d776a7612e44a50a3d8dd8418c747ba as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9d776a7612e44a50a3d8dd8418c747ba 2023-03-26 11:55:10,229 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9d776a7612e44a50a3d8dd8418c747ba, entries=150, sequenceid=108, filesize=11.6 K 2023-03-26 11:55:10,230 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/06356a2cbbae4d34b7224e45f1be0389 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/06356a2cbbae4d34b7224e45f1be0389 2023-03-26 11:55:10,233 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/06356a2cbbae4d34b7224e45f1be0389, entries=150, sequenceid=108, filesize=11.6 K 2023-03-26 11:55:10,233 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 863ms, sequenceid=108, compaction requested=true 2023-03-26 11:55:10,233 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:10,233 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:10,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:10,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,235 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,235 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:10,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25b3c3ca[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:10,237 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:10,237 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,238 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:10,238 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:10,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:10,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:10,240 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=93.93 KB heapSize=265.45 KB 2023-03-26 11:55:10,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:10,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:10,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:10,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:10,240 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:10,240 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:10,240 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,240 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:10,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:10,246 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:10,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:10,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:10,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:10,246 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:10,247 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,247 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,247 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,252 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,252 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:10,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:10,252 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,252 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:10,253 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:10,253 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,253 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:10,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831770301, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831770302, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831770303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831770311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,321 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831770312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,353 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:10,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:10,355 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:10,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:10,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:10,357 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:10,357 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,357 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:10,357 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:10,357 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,357 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,358 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:10,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,360 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:10,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:10,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:10,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,360 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:10,361 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,361 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:10,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:10,368 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,368 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:10,371 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53f48068[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:10,371 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:10,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,373 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:10,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:10,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:10,373 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:10,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:10,374 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:10,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:10,377 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:10,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:10,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:10,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:10,377 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:10,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:10,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:10,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,389 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:10,389 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:10,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,389 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:10,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831770413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831770413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,417 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,418 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831770413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831770417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,426 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831770422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,489 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:10,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:10,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:10,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:10,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:10,492 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:10,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:10,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:10,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:10,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:10,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:10,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:10,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:10,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:10,495 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,495 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:10,496 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,496 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:10,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:10,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:10,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@695c6c26[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:10,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:10,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,501 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:10,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:10,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:10,503 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:10,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:10,505 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:10,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:10,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:10,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:10,505 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:10,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,515 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,515 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,515 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:10,515 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:10,516 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:10,515 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,516 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:10,516 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:10,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,516 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,616 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:10,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:10,618 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:10,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:10,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:10,619 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:10,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:10,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:10,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,622 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,622 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,622 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:10,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,623 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:10,623 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:10,623 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:10,623 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:10,624 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,624 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:10,624 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,624 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:10,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:10,625 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,625 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831770618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:10,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831770619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,626 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831770620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:10,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16655543[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:10,627 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:10,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831770623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,629 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,629 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:10,629 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:10,629 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:10,629 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,629 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:10,632 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,632 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:10,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:10,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:10,634 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:10,634 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:10,634 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:10,634 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:10,634 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:10,636 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,636 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,636 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831770627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:10,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:10,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,640 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:10,640 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:10,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:10,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,665 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ba36184b28a14759a292a66177bc5ed7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ba36184b28a14759a292a66177bc5ed7 2023-03-26 11:55:10,740 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:10,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:10,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:10,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:10,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:10,744 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:10,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:10,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:10,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,746 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,746 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,746 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,746 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:10,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,746 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,747 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:10,747 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:10,747 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:10,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:10,747 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,748 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:10,748 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,748 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:10,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:10,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,750 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:10,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:10,752 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d7379d6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:10,752 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:10,752 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,753 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:10,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:10,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:10,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:10,755 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,755 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,755 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,755 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:10,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:10,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:10,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:10,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:10,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:10,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:10,758 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,758 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,758 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,759 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,760 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,763 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,763 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:10,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:10,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,764 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:10,764 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:10,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,864 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:10,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:10,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:10,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:10,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:10,867 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:10,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:10,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:10,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,869 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,869 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,869 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:10,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,870 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:10,870 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:10,870 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:10,870 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,870 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:10,870 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,871 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:10,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:10,872 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,873 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:10,874 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a734bd0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:10,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:10,874 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:10,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:10,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:10,876 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:10,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:10,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:10,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:10,883 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:10,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:10,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:10,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:10,883 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:10,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:10,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:10,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:10,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,886 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:10,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:10,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:10,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,889 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:10,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:10,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:10,892 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:10,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:10,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:10,892 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:10,893 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:10,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831770928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,931 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831770928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831770929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,937 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831770929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:10,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831770938, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,993 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:10,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:10,995 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:10,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:10,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:10,996 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:10,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:10,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:10,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,997 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:10,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:10,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:10,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:10,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:10,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:10,999 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:10,999 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:10,999 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,999 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:10,999 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:10,999 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:10,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,002 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50c431e4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,005 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,005 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,009 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,009 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,016 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:11,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,016 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,016 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,016 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,070 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=139, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/68d2a2b29d7f4943a80a6cee091372e8 2023-03-26 11:55:11,116 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,120 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,121 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,122 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,122 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,123 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,125 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cb7b9f2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,128 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,130 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,130 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,130 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,130 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,133 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,133 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,135 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,135 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,135 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,136 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,139 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,140 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,240 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,243 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,244 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,252 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,253 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,253 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,253 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,254 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,254 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,254 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,254 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5abcd07b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,260 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,260 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,260 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,261 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,261 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,261 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,262 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,265 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,265 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,265 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,265 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,265 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,265 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,274 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,274 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,274 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:11,274 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,274 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,274 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,274 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,374 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,378 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,378 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,378 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,378 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,378 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,379 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,382 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,382 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ad13d37[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,398 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,398 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,399 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,401 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,404 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,404 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,404 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,438 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:11,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831771433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:11,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831771436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:11,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831771439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,448 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:11,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831771441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:11,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831771447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,480 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/823829db3aad4dbfac34daf9c0855949 2023-03-26 11:55:11,505 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,508 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,516 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,516 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,517 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,517 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,517 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,517 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,520 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,520 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,520 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,520 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,522 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,525 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6beb6498[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,527 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,536 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,536 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,536 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,537 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,539 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,539 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,539 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,540 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,546 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,549 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,549 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,550 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,550 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,550 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,550 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,651 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,654 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,657 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,658 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,658 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,658 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,658 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,663 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,663 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,663 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,665 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ac2383e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,666 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,666 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,668 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,668 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,670 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,670 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,670 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,672 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,677 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,677 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,678 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,678 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,778 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,781 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,783 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,785 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,785 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,786 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,786 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,786 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,786 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,788 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6410815[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,791 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,793 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,797 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,797 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,798 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,798 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,798 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,802 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,804 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,804 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,804 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,894 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=139 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/f9e490e75776447db4352bda688a42bc 2023-03-26 11:55:11,900 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/68d2a2b29d7f4943a80a6cee091372e8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/68d2a2b29d7f4943a80a6cee091372e8 2023-03-26 11:55:11,904 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/68d2a2b29d7f4943a80a6cee091372e8, entries=250, sequenceid=139, filesize=38.9 K 2023-03-26 11:55:11,905 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:11,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/823829db3aad4dbfac34daf9c0855949 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/823829db3aad4dbfac34daf9c0855949 2023-03-26 11:55:11,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:11,909 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:11,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:11,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:11,911 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:11,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:11,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:11,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/823829db3aad4dbfac34daf9c0855949, entries=150, sequenceid=139, filesize=11.8 K 2023-03-26 11:55:11,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,912 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/f9e490e75776447db4352bda688a42bc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/f9e490e75776447db4352bda688a42bc 2023-03-26 11:55:11,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:11,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:11,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:11,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:11,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:11,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:11,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:11,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:11,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:11,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:11,917 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/f9e490e75776447db4352bda688a42bc, entries=150, sequenceid=139, filesize=11.8 K 2023-03-26 11:55:11,917 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~265.41 KB/271776, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 1677ms, sequenceid=139, compaction requested=true 2023-03-26 11:55:11,917 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:11,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:11,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:11,917 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 11:55:11,917 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:11,918 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:11,918 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:11,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:11,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15eaa56f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:11,920 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:11,920 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 175550 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:11,921 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:11,921 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): b57c4d4a48d50ba171b46d52c705a057/A is initiating minor compaction (all files) 2023-03-26 11:55:11,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,921 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b57c4d4a48d50ba171b46d52c705a057/A in TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,921 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): b57c4d4a48d50ba171b46d52c705a057/B is initiating minor compaction (all files) 2023-03-26 11:55:11,921 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b57c4d4a48d50ba171b46d52c705a057/B in TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:11,921 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8499a7bcd837430aa09146c0c822a7d2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/bd97644b2b034450b351bf2bcf1bb9f9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/ee94f970e44a4e33bdcbc76d908494e9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/149e57efef264a8eb1f7df4e74881c7a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5662b38139074e14819a354060c97405, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/68d2a2b29d7f4943a80a6cee091372e8] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp, totalSize=171.4 K 2023-03-26 11:55:11,921 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1db86fe3044845b39f63e1c4681cdba9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b60c6f4bbf604dcc972049afbd220939, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1b826e2732984a25a0db009d4e7910bb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9d776a7612e44a50a3d8dd8418c747ba, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/823829db3aad4dbfac34daf9c0855949] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp, totalSize=69.8 K 2023-03-26 11:55:11,921 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8499a7bcd837430aa09146c0c822a7d2, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679831704503 2023-03-26 11:55:11,921 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b2c5f1ba38ab4ea8bd0c5066c0aa74d4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679831704503 2023-03-26 11:55:11,922 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting bd97644b2b034450b351bf2bcf1bb9f9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679831704574 2023-03-26 11:55:11,922 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1db86fe3044845b39f63e1c4681cdba9, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679831704574 2023-03-26 11:55:11,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,922 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ee94f970e44a4e33bdcbc76d908494e9, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679831705705 2023-03-26 11:55:11,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,922 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b60c6f4bbf604dcc972049afbd220939, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679831705705 2023-03-26 11:55:11,922 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:11,922 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:11,922 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:11,922 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 149e57efef264a8eb1f7df4e74881c7a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679831706877 2023-03-26 11:55:11,922 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,923 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:11,923 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1b826e2732984a25a0db009d4e7910bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679831706877 2023-03-26 11:55:11,923 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9d776a7612e44a50a3d8dd8418c747ba, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1679831708020 2023-03-26 11:55:11,923 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5662b38139074e14819a354060c97405, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1679831708020 2023-03-26 11:55:11,923 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 823829db3aad4dbfac34daf9c0855949, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679831710229 2023-03-26 11:55:11,923 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 68d2a2b29d7f4943a80a6cee091372e8, keycount=250, bloomtype=ROW, size=38.9 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679831710213 2023-03-26 11:55:11,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:11,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:11,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:11,928 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:11,928 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:11,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:11,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:11,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:11,928 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:11,930 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,930 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,930 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:11,931 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,931 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:11,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:11,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:11,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:11,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:11,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:11,938 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:11,938 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:11,938 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:11,938 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:11,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:11,938 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:11,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:11,939 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): b57c4d4a48d50ba171b46d52c705a057#B#compaction#354 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:11,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:11,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:11,951 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): b57c4d4a48d50ba171b46d52c705a057#A#compaction#353 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:12,001 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/67396f01e67c4d049d1fc303140c5b92 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/67396f01e67c4d049d1fc303140c5b92 2023-03-26 11:55:12,005 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b57c4d4a48d50ba171b46d52c705a057/B of b57c4d4a48d50ba171b46d52c705a057 into 67396f01e67c4d049d1fc303140c5b92(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:12,006 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:12,006 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., storeName=b57c4d4a48d50ba171b46d52c705a057/B, priority=10, startTime=1679831711917; duration=0sec 2023-03-26 11:55:12,006 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:12,006 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:12,008 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71466 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:12,009 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): b57c4d4a48d50ba171b46d52c705a057/C is initiating minor compaction (all files) 2023-03-26 11:55:12,009 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b57c4d4a48d50ba171b46d52c705a057/C in TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:12,009 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/ce158224643b497f8b96732ad86b16a1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/22ba5aaf9ccd49c6860967cafeeea388, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/98aefd7e9bd847cb9fbc432d7d09b2da, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/417818068c224b42b927ec7ff4b40c94, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/06356a2cbbae4d34b7224e45f1be0389, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/f9e490e75776447db4352bda688a42bc] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp, totalSize=69.8 K 2023-03-26 11:55:12,009 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ce158224643b497f8b96732ad86b16a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=14, earliestPutTs=1679831704503 2023-03-26 11:55:12,010 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 22ba5aaf9ccd49c6860967cafeeea388, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1679831704574 2023-03-26 11:55:12,010 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 98aefd7e9bd847cb9fbc432d7d09b2da, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=61, earliestPutTs=1679831705705 2023-03-26 11:55:12,011 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 417818068c224b42b927ec7ff4b40c94, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1679831706877 2023-03-26 11:55:12,011 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 06356a2cbbae4d34b7224e45f1be0389, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=108, earliestPutTs=1679831708020 2023-03-26 11:55:12,011 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f9e490e75776447db4352bda688a42bc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679831710229 2023-03-26 11:55:12,025 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): b57c4d4a48d50ba171b46d52c705a057#C#compaction#355 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:12,038 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:12,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:12,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:12,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:12,044 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:12,044 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:12,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:12,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:12,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:12,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:12,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:12,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,046 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:12,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:12,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:12,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:12,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:12,046 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:12,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:12,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:12,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:12,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:12,047 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:12,047 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:12,048 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:12,048 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:12,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:12,445 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:12,457 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/bbb7a7ce368c40e8878c6a1342268352 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/bbb7a7ce368c40e8878c6a1342268352 2023-03-26 11:55:12,462 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b57c4d4a48d50ba171b46d52c705a057/C of b57c4d4a48d50ba171b46d52c705a057 into bbb7a7ce368c40e8878c6a1342268352(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:12,462 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:12,462 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., storeName=b57c4d4a48d50ba171b46d52c705a057/C, priority=10, startTime=1679831711917; duration=0sec 2023-03-26 11:55:12,462 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:12,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831772467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,476 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831772467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831772473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,481 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032636c61fe18b014b5fbf52bbf6b39440f4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032636c61fe18b014b5fbf52bbf6b39440f4 2023-03-26 11:55:12,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831772475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831772476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831772577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831772577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831772583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831772586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,590 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831772586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831772782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831772782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831772790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,800 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831772791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,803 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:12,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831772793, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:12,888 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=156, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/8cc7cc59b84540dd98551a9519854dbd 2023-03-26 11:55:13,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831773084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,095 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831773091, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831773101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831773103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,108 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831773104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,221 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/f84a91d2434c4dd3b1f09cc9cdcb4f5f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/f84a91d2434c4dd3b1f09cc9cdcb4f5f 2023-03-26 11:55:13,226 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b57c4d4a48d50ba171b46d52c705a057/A of b57c4d4a48d50ba171b46d52c705a057 into f84a91d2434c4dd3b1f09cc9cdcb4f5f(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-03-26 11:55:13,226 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:13,226 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., storeName=b57c4d4a48d50ba171b46d52c705a057/A, priority=10, startTime=1679831711917; duration=1sec 2023-03-26 11:55:13,226 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:13,298 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/d988b10daf6545c7a719e9421d15a696 2023-03-26 11:55:13,323 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=156 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/80efc585c52044f69c3325bcfd5912a7 2023-03-26 11:55:13,331 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/8cc7cc59b84540dd98551a9519854dbd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8cc7cc59b84540dd98551a9519854dbd 2023-03-26 11:55:13,338 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8cc7cc59b84540dd98551a9519854dbd, entries=150, sequenceid=156, filesize=25.3 K 2023-03-26 11:55:13,339 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/d988b10daf6545c7a719e9421d15a696 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d988b10daf6545c7a719e9421d15a696 2023-03-26 11:55:13,355 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d988b10daf6545c7a719e9421d15a696, entries=150, sequenceid=156, filesize=11.8 K 2023-03-26 11:55:13,356 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/80efc585c52044f69c3325bcfd5912a7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/80efc585c52044f69c3325bcfd5912a7 2023-03-26 11:55:13,365 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/80efc585c52044f69c3325bcfd5912a7, entries=150, sequenceid=156, filesize=11.8 K 2023-03-26 11:55:13,366 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 1318ms, sequenceid=156, compaction requested=false 2023-03-26 11:55:13,366 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:13,366 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:13,366 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:13,368 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,368 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:13,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:13,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:13,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:13,377 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2515ccee[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:13,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:13,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:13,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:13,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:13,378 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,379 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:13,384 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:13,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:13,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:13,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:13,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:13,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:13,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:13,389 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:13,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:13,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:13,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:13,389 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:13,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:13,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:13,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:13,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:13,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:13,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:13,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:13,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:13,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:13,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:13,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:13,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:13,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:13,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:13,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:13,395 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:13,396 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,396 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,396 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:13,396 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:13,396 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:13,396 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,496 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:13,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:13,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:13,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:13,499 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:13,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:13,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:13,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:13,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:13,501 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:13,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:13,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:13,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:13,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:13,502 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:13,502 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:13,502 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:13,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:13,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:13,503 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:13,503 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:13,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:13,594 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:13,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831773617, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831773618, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831773619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831773619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,631 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831773620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831773726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831773726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831773732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,734 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831773732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,738 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831773732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,921 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032600eab2b4497546ebb79aa5afb12a7cc3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032600eab2b4497546ebb79aa5afb12a7cc3 2023-03-26 11:55:13,936 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831773932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831773933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831773935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,938 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831773935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:13,942 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:13,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831773940, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,238 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831774237, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831774239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,243 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831774239, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831774240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,249 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831774245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,329 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/34c3e3c11cb445aa9c000c90822aafeb 2023-03-26 11:55:14,346 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/d949db37aaa34aba9967bf2d05f7c421 2023-03-26 11:55:14,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831774746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,751 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,751 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831774746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831774748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,752 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831774748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,753 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:14,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831774752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,757 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/1e5cb0d8c148433f8de9b459375f7ba2 2023-03-26 11:55:14,760 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/34c3e3c11cb445aa9c000c90822aafeb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/34c3e3c11cb445aa9c000c90822aafeb 2023-03-26 11:55:14,764 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/34c3e3c11cb445aa9c000c90822aafeb, entries=150, sequenceid=174, filesize=25.3 K 2023-03-26 11:55:14,764 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/d949db37aaa34aba9967bf2d05f7c421 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d949db37aaa34aba9967bf2d05f7c421 2023-03-26 11:55:14,768 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d949db37aaa34aba9967bf2d05f7c421, entries=150, sequenceid=174, filesize=11.8 K 2023-03-26 11:55:14,768 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/1e5cb0d8c148433f8de9b459375f7ba2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1e5cb0d8c148433f8de9b459375f7ba2 2023-03-26 11:55:14,771 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1e5cb0d8c148433f8de9b459375f7ba2, entries=150, sequenceid=174, filesize=11.8 K 2023-03-26 11:55:14,772 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 1269ms, sequenceid=174, compaction requested=true 2023-03-26 11:55:14,772 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:14,772 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:14,772 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:14,775 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:14,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:14,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:14,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e3d03ba[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:14,777 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:14,777 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,778 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:14,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:14,778 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,778 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:14,778 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:14,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:14,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:14,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:14,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:14,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:14,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:14,782 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:14,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:14,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:14,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:14,782 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:14,783 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,783 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,784 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:14,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:14,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:14,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:14,786 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:14,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:14,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:14,789 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:14,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:14,789 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:14,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:14,789 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:14,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:14,789 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:14,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,889 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:14,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:14,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:14,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:14,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:14,893 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:14,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:14,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:14,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:14,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,893 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:14,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,894 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:14,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:14,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:14,895 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:14,895 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:14,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:14,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:14,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:14,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:14,896 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:14,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:14,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:14,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:14,897 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:14,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:15,314 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267b5d6991d5e54b439d1f3ed0d69a94ed to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267b5d6991d5e54b439d1f3ed0d69a94ed 2023-03-26 11:55:15,720 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/372f4d44fe6740dd98c2e760b4fe8d47 2023-03-26 11:55:15,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:15,758 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:15,776 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,776 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831775768, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831775769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831775771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831775772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,778 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831775776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831775877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831775877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,882 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831775877, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831775879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:15,883 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:15,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831775879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831776083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,090 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831776083, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831776084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,091 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831776084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,091 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831776085, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,130 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/0bdaa12f97f541edbf2a9d49a31b9409 2023-03-26 11:55:16,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831776392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831776392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,396 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831776393, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831776394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,397 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831776394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,540 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/41268a48507041d18b2b1c2dbae185cb 2023-03-26 11:55:16,544 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/372f4d44fe6740dd98c2e760b4fe8d47 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/372f4d44fe6740dd98c2e760b4fe8d47 2023-03-26 11:55:16,547 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/372f4d44fe6740dd98c2e760b4fe8d47, entries=150, sequenceid=190, filesize=25.3 K 2023-03-26 11:55:16,548 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/0bdaa12f97f541edbf2a9d49a31b9409 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/0bdaa12f97f541edbf2a9d49a31b9409 2023-03-26 11:55:16,551 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/0bdaa12f97f541edbf2a9d49a31b9409, entries=150, sequenceid=190, filesize=11.8 K 2023-03-26 11:55:16,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/41268a48507041d18b2b1c2dbae185cb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/41268a48507041d18b2b1c2dbae185cb 2023-03-26 11:55:16,554 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/41268a48507041d18b2b1c2dbae185cb, entries=150, sequenceid=190, filesize=11.8 K 2023-03-26 11:55:16,555 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 1658ms, sequenceid=190, compaction requested=true 2023-03-26 11:55:16,555 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:16,555 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:16,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:16,558 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,558 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:16,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:16,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:16,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:16,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b2a3c8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:16,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:16,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:16,564 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:16,564 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:16,566 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:16,566 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,566 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,566 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:16,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:16,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:16,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:16,568 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:16,568 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:16,568 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:16,568 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:16,568 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:16,570 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,570 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,570 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:16,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:16,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:16,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:16,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:16,572 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:16,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:16,575 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:16,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:16,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:16,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:16,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:16,575 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:16,575 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:16,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:16,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:16,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,676 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:16,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:16,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:16,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:16,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:16,679 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:16,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:16,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:16,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:16,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:16,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:16,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:16,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:16,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:16,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:16,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:16,682 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:16,682 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:16,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:16,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:16,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:16,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:16,682 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:16,683 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:16,683 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:16,683 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:16,683 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:16,683 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:16,683 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:16,697 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303266335f3c68dad44f2829f40ff9020b7e7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303266335f3c68dad44f2829f40ff9020b7e7 2023-03-26 11:55:16,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:16,901 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:16,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831776923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,928 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831776924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831776925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831776927, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:16,934 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:16,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831776928, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831777029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,034 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831777029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831777034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831777034, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831777035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,102 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=206, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/df34b9efc2c44aa1a49ca749957cea55 2023-03-26 11:55:17,237 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831777235, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,241 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831777236, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831777240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831777240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,244 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831777241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,512 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/12880238fab34d2786accc4d5ba79039 2023-03-26 11:55:17,523 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=206 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/346c984b125e442c995c67c56b27ebe3 2023-03-26 11:55:17,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/df34b9efc2c44aa1a49ca749957cea55 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/df34b9efc2c44aa1a49ca749957cea55 2023-03-26 11:55:17,531 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/df34b9efc2c44aa1a49ca749957cea55, entries=150, sequenceid=206, filesize=25.3 K 2023-03-26 11:55:17,532 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/12880238fab34d2786accc4d5ba79039 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/12880238fab34d2786accc4d5ba79039 2023-03-26 11:55:17,540 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/12880238fab34d2786accc4d5ba79039, entries=150, sequenceid=206, filesize=11.8 K 2023-03-26 11:55:17,540 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/346c984b125e442c995c67c56b27ebe3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/346c984b125e442c995c67c56b27ebe3 2023-03-26 11:55:17,543 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/346c984b125e442c995c67c56b27ebe3, entries=150, sequenceid=206, filesize=11.8 K 2023-03-26 11:55:17,544 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 862ms, sequenceid=206, compaction requested=true 2023-03-26 11:55:17,544 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:17,544 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:17,544 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:17,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,547 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,548 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,548 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,548 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 11:55:17,548 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,548 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:17,548 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:17,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,549 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:17,549 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:17,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:17,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4033af6c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:17,551 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:17,552 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,553 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,553 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:17,553 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:17,553 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:17,554 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,554 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:17,554 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,554 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:17,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,555 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:17,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:17,561 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:17,561 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:17,561 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:17,561 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:17,561 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:17,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,574 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,574 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,574 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:17,575 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:17,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,575 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:17,577 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267f358ed7de294970b5629f1a038e273c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267f358ed7de294970b5629f1a038e273c 2023-03-26 11:55:17,578 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831777568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831777569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831777570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,580 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831777571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,581 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831777571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,588 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=223, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/de2061d275ad4c81ad2d7db79f28220c 2023-03-26 11:55:17,675 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:17,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:17,677 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:17,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:17,678 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:17,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:17,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:17,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:17,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,680 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,680 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:17,681 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,681 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:17,681 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:17,682 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:17,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:17,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:17,682 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:17,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:17,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831777679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,684 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,684 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:17,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:17,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bca7938[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:17,686 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:17,686 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,687 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:17,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:17,688 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:17,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831777681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831777681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831777682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,689 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831777682, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,689 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,689 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:17,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:17,692 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:17,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:17,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:17,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:17,692 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:17,693 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,696 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,699 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:17,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:17,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,700 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:17,700 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:17,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,800 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:17,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:17,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:17,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:17,803 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:17,803 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:17,803 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:17,803 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:17,803 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:17,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,807 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:17,807 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:17,807 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:17,807 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:17,808 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,808 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:17,808 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:17,808 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:17,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:17,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,810 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:17,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0c8328[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:17,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:17,812 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,813 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,813 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:17,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:17,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,813 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:17,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,814 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:17,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:17,816 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:17,816 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:17,816 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:17,817 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:17,817 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:17,819 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,819 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,819 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,821 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,823 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,823 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:17,824 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:17,823 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,824 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:17,824 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:17,895 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60154 deadline: 1679831777886, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831777889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831777890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,896 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831777890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,897 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:17,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831777891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,924 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:17,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:17,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:17,928 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:17,928 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:17,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:17,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:17,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:17,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,930 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,930 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,930 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,930 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:17,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:17,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,931 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:17,931 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:17,931 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:17,931 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:17,931 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:17,931 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:17,932 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:17,932 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:17,933 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,933 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:17,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38b5d9a2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:17,935 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:17,935 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:17,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:17,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:17,940 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,940 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:17,941 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:17,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,941 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:17,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:17,943 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:17,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:17,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:17,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:17,943 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:17,945 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,945 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:17,945 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:17,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:17,945 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:17,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:17,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:17,956 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:17,956 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:17,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:17,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,956 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:17,956 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:17,957 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:17,957 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:17,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:17,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,001 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/78792a9bd2c549898e3ff27d7bfa7fd2 2023-03-26 11:55:18,057 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:18,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:18,058 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:18,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:18,060 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:18,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:18,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:18,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,060 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:18,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:18,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,063 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:18,063 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:18,063 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:18,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:18,063 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,063 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:18,063 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:18,063 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:18,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:18,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:18,064 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:18,065 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:18,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:18,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65a04917[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:18,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,068 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:18,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:18,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:18,070 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:18,073 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:18,073 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:18,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:18,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:18,073 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:18,073 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:18,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:18,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,075 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,076 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:18,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:18,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:18,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:18,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:18,079 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:18,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,079 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:18,179 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:18,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:18,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:18,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:18,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:18,182 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:18,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:18,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:18,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,184 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:18,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,185 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:18,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:18,186 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,186 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:18,186 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:18,186 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:18,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7dbbef22[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:18,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:18,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,191 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:18,191 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:18,191 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,191 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:18,193 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:18,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:18,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:18,196 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:18,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:18,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:18,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:18,196 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:18,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,202 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:18,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,202 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:18,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:18,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,202 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:18,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:18,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:18,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,203 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:18,203 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:18,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831778197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831778197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,204 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831778197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831778198, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831778199, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,303 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:18,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:18,305 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:18,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:18,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:18,306 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:18,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:18,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:18,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:18,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,310 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:18,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,310 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:18,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:18,310 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:18,310 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:18,313 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:18,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:18,314 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3467267e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:18,314 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:18,314 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:18,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:18,316 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:18,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:18,317 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:18,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:18,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:18,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:18,319 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:18,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:18,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:18,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:18,320 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:18,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:18,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:18,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:18,323 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,323 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:18,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:18,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:18,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:18,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:18,331 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:18,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:18,332 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:18,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,332 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:18,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:18,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:18,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:18,411 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=223 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/1afe82d74fcb4d4bb7345ccf7c9df990 2023-03-26 11:55:18,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/de2061d275ad4c81ad2d7db79f28220c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/de2061d275ad4c81ad2d7db79f28220c 2023-03-26 11:55:18,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/de2061d275ad4c81ad2d7db79f28220c, entries=200, sequenceid=223, filesize=32.1 K 2023-03-26 11:55:18,420 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/78792a9bd2c549898e3ff27d7bfa7fd2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/78792a9bd2c549898e3ff27d7bfa7fd2 2023-03-26 11:55:18,423 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/78792a9bd2c549898e3ff27d7bfa7fd2, entries=150, sequenceid=223, filesize=11.8 K 2023-03-26 11:55:18,423 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/1afe82d74fcb4d4bb7345ccf7c9df990 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1afe82d74fcb4d4bb7345ccf7c9df990 2023-03-26 11:55:18,426 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1afe82d74fcb4d4bb7345ccf7c9df990, entries=150, sequenceid=223, filesize=11.8 K 2023-03-26 11:55:18,427 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 879ms, sequenceid=223, compaction requested=true 2023-03-26 11:55:18,427 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:18,427 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:18,427 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:18,427 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:18,427 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:18,427 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:18,429 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162572 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:18,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72422 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:18,429 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): b57c4d4a48d50ba171b46d52c705a057/A is initiating minor compaction (all files) 2023-03-26 11:55:18,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): b57c4d4a48d50ba171b46d52c705a057/B is initiating minor compaction (all files) 2023-03-26 11:55:18,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of b57c4d4a48d50ba171b46d52c705a057/A in TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b57c4d4a48d50ba171b46d52c705a057/B in TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/f84a91d2434c4dd3b1f09cc9cdcb4f5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8cc7cc59b84540dd98551a9519854dbd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/34c3e3c11cb445aa9c000c90822aafeb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/372f4d44fe6740dd98c2e760b4fe8d47, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/df34b9efc2c44aa1a49ca749957cea55, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/de2061d275ad4c81ad2d7db79f28220c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp, totalSize=158.8 K 2023-03-26 11:55:18,430 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/67396f01e67c4d049d1fc303140c5b92, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d988b10daf6545c7a719e9421d15a696, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d949db37aaa34aba9967bf2d05f7c421, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/0bdaa12f97f541edbf2a9d49a31b9409, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/12880238fab34d2786accc4d5ba79039, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/78792a9bd2c549898e3ff27d7bfa7fd2] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp, totalSize=70.7 K 2023-03-26 11:55:18,430 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f84a91d2434c4dd3b1f09cc9cdcb4f5f, keycount=150, bloomtype=ROW, size=25.5 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679831710229 2023-03-26 11:55:18,430 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 67396f01e67c4d049d1fc303140c5b92, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679831710229 2023-03-26 11:55:18,430 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8cc7cc59b84540dd98551a9519854dbd, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679831710300 2023-03-26 11:55:18,430 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d988b10daf6545c7a719e9421d15a696, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679831710300 2023-03-26 11:55:18,430 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 34c3e3c11cb445aa9c000c90822aafeb, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679831712466 2023-03-26 11:55:18,431 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d949db37aaa34aba9967bf2d05f7c421, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679831712466 2023-03-26 11:55:18,431 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 372f4d44fe6740dd98c2e760b4fe8d47, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679831713609 2023-03-26 11:55:18,431 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 0bdaa12f97f541edbf2a9d49a31b9409, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679831713609 2023-03-26 11:55:18,431 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 12880238fab34d2786accc4d5ba79039, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679831715763 2023-03-26 11:55:18,432 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting df34b9efc2c44aa1a49ca749957cea55, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679831715763 2023-03-26 11:55:18,432 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 78792a9bd2c549898e3ff27d7bfa7fd2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679831716923 2023-03-26 11:55:18,432 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:18,432 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting de2061d275ad4c81ad2d7db79f28220c, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679831716920 2023-03-26 11:55:18,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:18,434 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:18,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:18,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:18,435 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:18,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:18,435 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:18,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,437 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:18,437 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:18,438 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:18,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:18,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:18,439 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:18,439 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:18,441 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,441 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:18,441 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:18,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:18,445 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): b57c4d4a48d50ba171b46d52c705a057#B#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:18,463 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): b57c4d4a48d50ba171b46d52c705a057#A#compaction#372 average throughput is 1.04 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:18,472 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/56aecdb8ca7d4e6e9042ba88c987c550 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/56aecdb8ca7d4e6e9042ba88c987c550 2023-03-26 11:55:18,479 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b57c4d4a48d50ba171b46d52c705a057/B of b57c4d4a48d50ba171b46d52c705a057 into 56aecdb8ca7d4e6e9042ba88c987c550(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:18,479 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:18,479 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., storeName=b57c4d4a48d50ba171b46d52c705a057/B, priority=10, startTime=1679831718427; duration=0sec 2023-03-26 11:55:18,480 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:18,480 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:18,481 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72422 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:18,481 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): b57c4d4a48d50ba171b46d52c705a057/C is initiating minor compaction (all files) 2023-03-26 11:55:18,482 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of b57c4d4a48d50ba171b46d52c705a057/C in TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:18,482 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/bbb7a7ce368c40e8878c6a1342268352, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/80efc585c52044f69c3325bcfd5912a7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1e5cb0d8c148433f8de9b459375f7ba2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/41268a48507041d18b2b1c2dbae185cb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/346c984b125e442c995c67c56b27ebe3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1afe82d74fcb4d4bb7345ccf7c9df990] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp, totalSize=70.7 K 2023-03-26 11:55:18,482 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bbb7a7ce368c40e8878c6a1342268352, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=139, earliestPutTs=1679831710229 2023-03-26 11:55:18,483 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 80efc585c52044f69c3325bcfd5912a7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=156, earliestPutTs=1679831710300 2023-03-26 11:55:18,483 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1e5cb0d8c148433f8de9b459375f7ba2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679831712466 2023-03-26 11:55:18,484 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 41268a48507041d18b2b1c2dbae185cb, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679831713609 2023-03-26 11:55:18,484 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 346c984b125e442c995c67c56b27ebe3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=206, earliestPutTs=1679831715763 2023-03-26 11:55:18,484 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1afe82d74fcb4d4bb7345ccf7c9df990, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=223, earliestPutTs=1679831716923 2023-03-26 11:55:18,505 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): b57c4d4a48d50ba171b46d52c705a057#C#compaction#374 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:18,513 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/82bf9813275d4957a585baa4fa4f3f66 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/82bf9813275d4957a585baa4fa4f3f66 2023-03-26 11:55:18,518 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b57c4d4a48d50ba171b46d52c705a057/C of b57c4d4a48d50ba171b46d52c705a057 into 82bf9813275d4957a585baa4fa4f3f66(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:18,518 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:18,518 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., storeName=b57c4d4a48d50ba171b46d52c705a057/C, priority=10, startTime=1679831718427; duration=0sec 2023-03-26 11:55:18,518 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:18,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:18,710 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:18,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831778734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831778735, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831778736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831778737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831778739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831778844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60184 deadline: 1679831778845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60120 deadline: 1679831778845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60126 deadline: 1679831778845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,850 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:18,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831778845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:18,902 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303264752c8229bfb4270b19036726ee32528 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264752c8229bfb4270b19036726ee32528 2023-03-26 11:55:18,929 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=239, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/b9d742d0dfe54a76a3c763fe531cdb6b 2023-03-26 11:55:18,946 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/59601ca254c447c1986fbeaed4a960f5 2023-03-26 11:55:19,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831779047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:60168 deadline: 1679831779047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831779048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,057 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831779051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,058 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831779052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,338 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/9385c626589e49a6a9c8d1ba1c55b1a5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9385c626589e49a6a9c8d1ba1c55b1a5 2023-03-26 11:55:19,343 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in b57c4d4a48d50ba171b46d52c705a057/A of b57c4d4a48d50ba171b46d52c705a057 into 9385c626589e49a6a9c8d1ba1c55b1a5(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:19,343 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:19,343 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057., storeName=b57c4d4a48d50ba171b46d52c705a057/A, priority=10, startTime=1679831718427; duration=0sec 2023-03-26 11:55:19,343 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:19,361 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831779353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831779353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,362 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831779354, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831779360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831779361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,377 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=239 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/822d3d378a6643358d07fe89f8629579 2023-03-26 11:55:19,381 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/b9d742d0dfe54a76a3c763fe531cdb6b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/b9d742d0dfe54a76a3c763fe531cdb6b 2023-03-26 11:55:19,385 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/b9d742d0dfe54a76a3c763fe531cdb6b, entries=150, sequenceid=239, filesize=25.3 K 2023-03-26 11:55:19,385 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/59601ca254c447c1986fbeaed4a960f5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/59601ca254c447c1986fbeaed4a960f5 2023-03-26 11:55:19,398 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/59601ca254c447c1986fbeaed4a960f5, entries=150, sequenceid=239, filesize=11.8 K 2023-03-26 11:55:19,398 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/822d3d378a6643358d07fe89f8629579 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/822d3d378a6643358d07fe89f8629579 2023-03-26 11:55:19,402 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/822d3d378a6643358d07fe89f8629579, entries=150, sequenceid=239, filesize=11.8 K 2023-03-26 11:55:19,402 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 961ms, sequenceid=239, compaction requested=false 2023-03-26 11:55:19,403 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:19,403 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:19,403 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:19,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:19,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:19,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22099199[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:19,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:19,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:19,409 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:19,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:19,411 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:19,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:19,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:19,414 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:19,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:19,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:19,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:19,414 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:19,416 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:19,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:19,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:19,417 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:19,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:19,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:19,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:19,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:19,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:19,422 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:19,422 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:19,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:19,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:19,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:19,522 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:19,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:19,525 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:19,526 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:19,526 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:19,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:19,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:19,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:19,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:19,528 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:19,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:19,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:19,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:19,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:19,529 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:19,529 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:19,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:19,530 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:19,530 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:19,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:19,868 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:19,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831779880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831779882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831779883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831779883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,890 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831779884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,944 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f2306281ea8b4b61b151bcdceab63674 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2306281ea8b4b61b151bcdceab63674 2023-03-26 11:55:19,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831779990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831779990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,994 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831779990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831779990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:19,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:19,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831779990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831780195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,199 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831780196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,200 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831780196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831780196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,201 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831780197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,356 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=258, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/50c77aa1c09144b58a644b0d8abe1ec7 2023-03-26 11:55:20,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831780500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831780501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,503 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831780502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831780504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,507 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:20,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831780504, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:20,766 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/21610e6106f04401a498659bb49b4b83 2023-03-26 11:55:21,008 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:21,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831781004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:21,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831781006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:21,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831781008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:21,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831781008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,015 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:21,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831781010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,183 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/16146a51686244618e9b87f88e56a76f 2023-03-26 11:55:21,187 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/50c77aa1c09144b58a644b0d8abe1ec7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/50c77aa1c09144b58a644b0d8abe1ec7 2023-03-26 11:55:21,190 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/50c77aa1c09144b58a644b0d8abe1ec7, entries=150, sequenceid=258, filesize=25.3 K 2023-03-26 11:55:21,191 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/21610e6106f04401a498659bb49b4b83 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/21610e6106f04401a498659bb49b4b83 2023-03-26 11:55:21,194 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/21610e6106f04401a498659bb49b4b83, entries=150, sequenceid=258, filesize=11.8 K 2023-03-26 11:55:21,194 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/16146a51686244618e9b87f88e56a76f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/16146a51686244618e9b87f88e56a76f 2023-03-26 11:55:21,197 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/16146a51686244618e9b87f88e56a76f, entries=150, sequenceid=258, filesize=11.8 K 2023-03-26 11:55:21,198 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for b57c4d4a48d50ba171b46d52c705a057 in 1668ms, sequenceid=258, compaction requested=true 2023-03-26 11:55:21,198 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:21,198 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:21,198 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:21,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:21,202 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45068c3e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:21,202 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:21,202 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:21,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:21,204 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,204 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:21,204 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:21,205 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:21,205 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:21,207 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:21,207 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:21,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:21,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:21,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:21,208 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:21,209 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,209 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,209 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:21,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:21,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:21,215 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:21,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:21,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:21,215 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:21,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:21,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:21,217 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:21,217 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:21,317 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:21,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:21,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:21,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:21,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:21,320 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:21,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:21,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:21,320 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:21,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,322 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:21,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,322 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:21,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,322 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:21,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,323 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:21,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,323 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:21,324 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:21,324 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:21,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:21,324 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:21,324 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:21,324 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:21,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:21,336 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326655442da5d354a59a26be30ae8589959 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326655442da5d354a59a26be30ae8589959 2023-03-26 11:55:21,741 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=274, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/9af3f4c125064950ad5b3d260e9dea01 2023-03-26 11:55:21,753 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/9461b833afa34706bc7aa4df18b15f20 2023-03-26 11:55:21,769 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/860abb228d9d472e9f11955b5399f9e4 2023-03-26 11:55:21,774 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/9af3f4c125064950ad5b3d260e9dea01 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9af3f4c125064950ad5b3d260e9dea01 2023-03-26 11:55:21,777 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9af3f4c125064950ad5b3d260e9dea01, entries=150, sequenceid=274, filesize=25.4 K 2023-03-26 11:55:21,778 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/9461b833afa34706bc7aa4df18b15f20 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9461b833afa34706bc7aa4df18b15f20 2023-03-26 11:55:21,782 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9461b833afa34706bc7aa4df18b15f20, entries=150, sequenceid=274, filesize=11.9 K 2023-03-26 11:55:21,782 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/860abb228d9d472e9f11955b5399f9e4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/860abb228d9d472e9f11955b5399f9e4 2023-03-26 11:55:21,787 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/860abb228d9d472e9f11955b5399f9e4, entries=150, sequenceid=274, filesize=11.9 K 2023-03-26 11:55:21,788 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for b57c4d4a48d50ba171b46d52c705a057 in 464ms, sequenceid=274, compaction requested=true 2023-03-26 11:55:21,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:21,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:21,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:21,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:21,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:21,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@67e6a943[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:21,792 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:21,792 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,793 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:21,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:21,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:21,795 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:21,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,796 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:21,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:21,799 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:21,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:21,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:21,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:21,799 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:21,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,800 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:21,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,802 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:21,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:21,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,805 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:21,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:21,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:21,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:21,806 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:21,806 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:21,806 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:21,906 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:21,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:21,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:21,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:21,910 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:21,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:21,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:21,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:21,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,911 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:21,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:21,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:21,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,912 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,913 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:21,913 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:21,913 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:21,913 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:21,914 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:21,914 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:21,914 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:21,914 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:21,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:21,919 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,919 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,920 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,920 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:21,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6abf3717[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:21,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:21,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,922 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:21,923 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,923 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:21,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:21,924 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:21,927 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:21,927 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:21,927 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:21,927 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:21,927 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:21,927 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:21,928 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,928 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,929 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:21,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,931 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:21,934 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:21,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:21,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:21,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:21,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:21,935 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:21,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:21,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,935 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:21,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:21,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:21,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:21,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:21,935 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:21,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:21,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:22,035 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:22,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:22,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:22,039 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:22,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:22,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:22,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:22,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:22,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:22,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:22,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:22,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:22,053 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:22,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:22,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:22,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:22,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:22,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:22,072 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:22,072 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:22,072 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:22,072 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-03-26 11:55:22,072 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:22,072 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:22,073 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:22,073 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:22,073 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:22,073 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:22,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:22,076 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:22,087 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326159d395f85ba4505b7aeb78624772538 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326159d395f85ba4505b7aeb78624772538 2023-03-26 11:55:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831722099, max timestamp=1679831722122Num uniques 859; 2023-03-26 11:55:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831722099, max timestamp=1679831722122Num uniques 859; 2023-03-26 11:55:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1679831722099, max timestamp=1679831722122Num uniques 859; 2023-03-26 11:55:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:55:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:55:22,125 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-03-26 11:55:22,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831782153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831782153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831782154, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831782155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831782165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831782270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831782271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831782271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831782271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,277 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831782271, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831782477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831782478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831782478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,484 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831782479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831782479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,494 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=283, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/9cd69a2f14ef4133b0ed3d9725d008cf 2023-03-26 11:55:22,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831782786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831782786, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,793 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831782787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831782787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:22,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831782787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:22,904 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/f93a1d0a236e455792d8f576a3e11f93 2023-03-26 11:55:23,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:23,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831783292, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:23,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831783295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:23,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831783295, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,301 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:23,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831783296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:23,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831783296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,313 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/db95b3a3022c4a25847f549a880de465 2023-03-26 11:55:23,317 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/9cd69a2f14ef4133b0ed3d9725d008cf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9cd69a2f14ef4133b0ed3d9725d008cf 2023-03-26 11:55:23,320 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9cd69a2f14ef4133b0ed3d9725d008cf, entries=250, sequenceid=283, filesize=39.2 K 2023-03-26 11:55:23,321 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/f93a1d0a236e455792d8f576a3e11f93 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/f93a1d0a236e455792d8f576a3e11f93 2023-03-26 11:55:23,324 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/f93a1d0a236e455792d8f576a3e11f93, entries=150, sequenceid=283, filesize=11.9 K 2023-03-26 11:55:23,325 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/db95b3a3022c4a25847f549a880de465 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/db95b3a3022c4a25847f549a880de465 2023-03-26 11:55:23,327 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/db95b3a3022c4a25847f549a880de465, entries=150, sequenceid=283, filesize=11.9 K 2023-03-26 11:55:23,328 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=154.31 KB/158010 for b57c4d4a48d50ba171b46d52c705a057 in 1256ms, sequenceid=283, compaction requested=true 2023-03-26 11:55:23,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:23,328 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:23,328 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:23,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:23,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:23,331 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454f680[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:23,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:23,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,333 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:23,333 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:23,333 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,333 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:23,334 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:23,334 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:23,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:23,337 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:23,337 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:23,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:23,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:23,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:23,337 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:23,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:23,340 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:23,341 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:23,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:23,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:23,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:23,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:23,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:23,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:23,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:23,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:23,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:23,344 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:23,344 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:23,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,344 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:23,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:23,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:23,344 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,444 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:23,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:23,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:23,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:23,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:23,448 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:23,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:23,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:23,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:23,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:23,450 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:23,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:23,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:23,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:23,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:23,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:23,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:23,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:23,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:23,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:23,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:23,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:23,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:23,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. started... 2023-03-26 11:55:23,452 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=154.31 KB heapSize=430.69 KB 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:23,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:23,453 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:23,867 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326c8a24104d11645c3b41f3c1e40fb7467 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c8a24104d11645c3b41f3c1e40fb7467 2023-03-26 11:55:24,272 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=320, memsize=51.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/5a42fe72615b41da95cfce3b6b586ed3 2023-03-26 11:55:24,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:24,308 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. as already flushing 2023-03-26 11:55:24,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831784310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831784311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60126 deadline: 1679831784311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,333 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831784319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831784320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831784425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,432 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831784425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831784434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831784435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,489 DEBUG [Thread-2515] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a7503d4 to 127.0.0.1:53955 2023-03-26 11:55:24,489 DEBUG [Thread-2515] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:24,494 DEBUG [Thread-2513] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x711ebaf2 to 127.0.0.1:53955 2023-03-26 11:55:24,494 DEBUG [Thread-2513] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:24,496 DEBUG [Thread-2517] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7eae4191 to 127.0.0.1:53955 2023-03-26 11:55:24,496 DEBUG [Thread-2517] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:24,498 DEBUG [Thread-2511] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f9e86d9 to 127.0.0.1:53955 2023-03-26 11:55:24,498 DEBUG [Thread-2511] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:24,503 DEBUG [Thread-2509] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x21128196 to 127.0.0.1:53955 2023-03-26 11:55:24,503 DEBUG [Thread-2509] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:24,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831784632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,634 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831784633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831784643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831784643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,682 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/2d249229b25b4e199f746aa0897fc1dc 2023-03-26 11:55:24,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60154 deadline: 1679831784936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,936 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60168 deadline: 1679831784936, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,944 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60184 deadline: 1679831784944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:24,946 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:24,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:60120 deadline: 1679831784946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,090 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=51.44 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/66bbaf7c4fc04fa6b5b617b2dff463a3 2023-03-26 11:55:25,093 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/5a42fe72615b41da95cfce3b6b586ed3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5a42fe72615b41da95cfce3b6b586ed3 2023-03-26 11:55:25,096 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5a42fe72615b41da95cfce3b6b586ed3, entries=150, sequenceid=320, filesize=25.4 K 2023-03-26 11:55:25,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/2d249229b25b4e199f746aa0897fc1dc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/2d249229b25b4e199f746aa0897fc1dc 2023-03-26 11:55:25,099 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/2d249229b25b4e199f746aa0897fc1dc, entries=150, sequenceid=320, filesize=11.9 K 2023-03-26 11:55:25,100 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/66bbaf7c4fc04fa6b5b617b2dff463a3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/66bbaf7c4fc04fa6b5b617b2dff463a3 2023-03-26 11:55:25,102 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/66bbaf7c4fc04fa6b5b617b2dff463a3, entries=150, sequenceid=320, filesize=11.9 K 2023-03-26 11:55:25,103 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~154.31 KB/158010, heapSize ~430.64 KB/440976, currentSize=33.54 KB/34350 for b57c4d4a48d50ba171b46d52c705a057 in 1651ms, sequenceid=320, compaction requested=true 2023-03-26 11:55:25,103 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:25,103 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:25,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,108 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:25,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:25,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:25,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75c2fd48[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:25,109 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:25,109 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:25,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:25,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:25,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:25,114 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:25,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:25,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:25,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:25,114 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:25,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:25,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:25,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,117 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:25,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:25,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:25,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:25,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:25,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:25,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:25,120 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:25,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:25,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:25,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,121 DEBUG [Thread-2508] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:25,121 DEBUG [Thread-2508] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:25,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:25,121 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:25,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:25,221 DEBUG [Thread-2508] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:25,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:25,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:25,439 DEBUG [Thread-2500] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4d010713 to 127.0.0.1:53955 2023-03-26 11:55:25,439 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:55:25,439 DEBUG [Thread-2500] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:25,439 DEBUG [Thread-2498] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x445919fd to 127.0.0.1:53955 2023-03-26 11:55:25,440 DEBUG [Thread-2498] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:25,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:25,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:25,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:25,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:25,440 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:25,440 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:25,447 DEBUG [Thread-2502] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x69100738 to 127.0.0.1:53955 2023-03-26 11:55:25,447 DEBUG [Thread-2502] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:25,449 DEBUG [Thread-2504] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f603c27 to 127.0.0.1:53955 2023-03-26 11:55:25,449 DEBUG [Thread-2504] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:25,849 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326dd5b3051bbb940299e5916976b844cb4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326dd5b3051bbb940299e5916976b844cb4 2023-03-26 11:55:26,254 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=330, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/20cba0ac4e924b55a0b16634388c1bc8 2023-03-26 11:55:26,261 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/461b4bf845a24569b5f21767469b7580 2023-03-26 11:55:26,340 DEBUG [Thread-2506] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e01d5e5 to 127.0.0.1:53955 2023-03-26 11:55:26,340 DEBUG [Thread-2506] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:26,340 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 11:55:26,340 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 48 2023-03-26 11:55:26,340 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 69 2023-03-26 11:55:26,340 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 51 2023-03-26 11:55:26,340 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 53 2023-03-26 11:55:26,340 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 50 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1837 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5510 rows 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1811 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5433 rows 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1827 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5481 rows 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1821 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5463 rows 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 1822 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 5466 rows 2023-03-26 11:55:26,341 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:55:26,341 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bb27fd4 to 127.0.0.1:53955 2023-03-26 11:55:26,341 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:26,343 INFO [Listener at localhost/46409] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 11:55:26,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 11:55:26,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 11:55:26,347 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831726347"}]},"ts":"1679831726347"} 2023-03-26 11:55:26,348 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 11:55:26,350 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 11:55:26,350 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, UNASSIGN}] 2023-03-26 11:55:26,352 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, UNASSIGN 2023-03-26 11:55:26,352 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:26,352 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831726352"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831726352"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831726352"}]},"ts":"1679831726352"} 2023-03-26 11:55:26,353 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:55:26,448 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 11:55:26,505 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:26,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing b57c4d4a48d50ba171b46d52c705a057, disabling compactions & flushes 2023-03-26 11:55:26,505 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1858): waiting for 0 compactions & cache flush to complete for region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 11:55:26,669 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=330 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/38631826874b4130a05e3f80f53c5782 2023-03-26 11:55:26,672 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/20cba0ac4e924b55a0b16634388c1bc8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/20cba0ac4e924b55a0b16634388c1bc8 2023-03-26 11:55:26,674 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/20cba0ac4e924b55a0b16634388c1bc8, entries=150, sequenceid=330, filesize=25.4 K 2023-03-26 11:55:26,675 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/461b4bf845a24569b5f21767469b7580 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/461b4bf845a24569b5f21767469b7580 2023-03-26 11:55:26,678 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/461b4bf845a24569b5f21767469b7580, entries=150, sequenceid=330, filesize=11.9 K 2023-03-26 11:55:26,678 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/38631826874b4130a05e3f80f53c5782 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/38631826874b4130a05e3f80f53c5782 2023-03-26 11:55:26,681 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/38631826874b4130a05e3f80f53c5782, entries=150, sequenceid=330, filesize=11.9 K 2023-03-26 11:55:26,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=20.13 KB/20610 for b57c4d4a48d50ba171b46d52c705a057 in 1242ms, sequenceid=330, compaction requested=true 2023-03-26 11:55:26,681 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:26,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:26,681 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. because compaction request was cancelled 2023-03-26 11:55:26,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:26,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:26,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. after waiting 0 ms 2023-03-26 11:55:26,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:26,681 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:26,681 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. because compaction request was cancelled 2023-03-26 11:55:26,681 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing b57c4d4a48d50ba171b46d52c705a057 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-03-26 11:55:26,681 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. because compaction request was cancelled 2023-03-26 11:55:26,681 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:26,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=A 2023-03-26 11:55:26,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:26,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=B 2023-03-26 11:55:26,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:26,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK b57c4d4a48d50ba171b46d52c705a057, store=C 2023-03-26 11:55:26,682 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:26,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 11:55:27,091 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326b035c7dbf76e45bd9d40ae23a174a900 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b035c7dbf76e45bd9d40ae23a174a900 2023-03-26 11:55:27,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 11:55:27,496 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=336, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/a525aff123954ecd9bfe19dc766bab96 2023-03-26 11:55:27,903 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/cf4c7cccb06e4be89d53fd81a6538120 2023-03-26 11:55:28,311 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=336 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/148d6bbe60be4d74801d1536f61f8571 2023-03-26 11:55:28,314 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/A/a525aff123954ecd9bfe19dc766bab96 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/a525aff123954ecd9bfe19dc766bab96 2023-03-26 11:55:28,317 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/a525aff123954ecd9bfe19dc766bab96, entries=100, sequenceid=336, filesize=18.6 K 2023-03-26 11:55:28,318 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/B/cf4c7cccb06e4be89d53fd81a6538120 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/cf4c7cccb06e4be89d53fd81a6538120 2023-03-26 11:55:28,321 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/cf4c7cccb06e4be89d53fd81a6538120, entries=100, sequenceid=336, filesize=9.5 K 2023-03-26 11:55:28,321 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/.tmp/C/148d6bbe60be4d74801d1536f61f8571 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/148d6bbe60be4d74801d1536f61f8571 2023-03-26 11:55:28,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/148d6bbe60be4d74801d1536f61f8571, entries=100, sequenceid=336, filesize=9.5 K 2023-03-26 11:55:28,324 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for b57c4d4a48d50ba171b46d52c705a057 in 1643ms, sequenceid=336, compaction requested=true 2023-03-26 11:55:28,336 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8499a7bcd837430aa09146c0c822a7d2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/bd97644b2b034450b351bf2bcf1bb9f9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/ee94f970e44a4e33bdcbc76d908494e9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/149e57efef264a8eb1f7df4e74881c7a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5662b38139074e14819a354060c97405, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/68d2a2b29d7f4943a80a6cee091372e8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/f84a91d2434c4dd3b1f09cc9cdcb4f5f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8cc7cc59b84540dd98551a9519854dbd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/34c3e3c11cb445aa9c000c90822aafeb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/372f4d44fe6740dd98c2e760b4fe8d47, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/df34b9efc2c44aa1a49ca749957cea55, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/de2061d275ad4c81ad2d7db79f28220c] to archive 2023-03-26 11:55:28,337 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:28,338 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8499a7bcd837430aa09146c0c822a7d2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8499a7bcd837430aa09146c0c822a7d2 2023-03-26 11:55:28,339 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/bd97644b2b034450b351bf2bcf1bb9f9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/bd97644b2b034450b351bf2bcf1bb9f9 2023-03-26 11:55:28,340 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/ee94f970e44a4e33bdcbc76d908494e9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/ee94f970e44a4e33bdcbc76d908494e9 2023-03-26 11:55:28,340 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/149e57efef264a8eb1f7df4e74881c7a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/149e57efef264a8eb1f7df4e74881c7a 2023-03-26 11:55:28,341 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5662b38139074e14819a354060c97405 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5662b38139074e14819a354060c97405 2023-03-26 11:55:28,342 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/68d2a2b29d7f4943a80a6cee091372e8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/68d2a2b29d7f4943a80a6cee091372e8 2023-03-26 11:55:28,343 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/f84a91d2434c4dd3b1f09cc9cdcb4f5f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/f84a91d2434c4dd3b1f09cc9cdcb4f5f 2023-03-26 11:55:28,344 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8cc7cc59b84540dd98551a9519854dbd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/8cc7cc59b84540dd98551a9519854dbd 2023-03-26 11:55:28,344 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/34c3e3c11cb445aa9c000c90822aafeb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/34c3e3c11cb445aa9c000c90822aafeb 2023-03-26 11:55:28,345 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/372f4d44fe6740dd98c2e760b4fe8d47 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/372f4d44fe6740dd98c2e760b4fe8d47 2023-03-26 11:55:28,346 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/df34b9efc2c44aa1a49ca749957cea55 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/df34b9efc2c44aa1a49ca749957cea55 2023-03-26 11:55:28,347 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/de2061d275ad4c81ad2d7db79f28220c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/de2061d275ad4c81ad2d7db79f28220c 2023-03-26 11:55:28,358 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1db86fe3044845b39f63e1c4681cdba9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b60c6f4bbf604dcc972049afbd220939, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1b826e2732984a25a0db009d4e7910bb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9d776a7612e44a50a3d8dd8418c747ba, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/67396f01e67c4d049d1fc303140c5b92, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/823829db3aad4dbfac34daf9c0855949, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d988b10daf6545c7a719e9421d15a696, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d949db37aaa34aba9967bf2d05f7c421, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/0bdaa12f97f541edbf2a9d49a31b9409, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/12880238fab34d2786accc4d5ba79039, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/78792a9bd2c549898e3ff27d7bfa7fd2] to archive 2023-03-26 11:55:28,359 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:28,360 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b2c5f1ba38ab4ea8bd0c5066c0aa74d4 2023-03-26 11:55:28,361 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1db86fe3044845b39f63e1c4681cdba9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1db86fe3044845b39f63e1c4681cdba9 2023-03-26 11:55:28,362 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b60c6f4bbf604dcc972049afbd220939 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/b60c6f4bbf604dcc972049afbd220939 2023-03-26 11:55:28,362 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1b826e2732984a25a0db009d4e7910bb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/1b826e2732984a25a0db009d4e7910bb 2023-03-26 11:55:28,363 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9d776a7612e44a50a3d8dd8418c747ba to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9d776a7612e44a50a3d8dd8418c747ba 2023-03-26 11:55:28,364 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/67396f01e67c4d049d1fc303140c5b92 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/67396f01e67c4d049d1fc303140c5b92 2023-03-26 11:55:28,365 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/823829db3aad4dbfac34daf9c0855949 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/823829db3aad4dbfac34daf9c0855949 2023-03-26 11:55:28,366 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d988b10daf6545c7a719e9421d15a696 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d988b10daf6545c7a719e9421d15a696 2023-03-26 11:55:28,366 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d949db37aaa34aba9967bf2d05f7c421 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/d949db37aaa34aba9967bf2d05f7c421 2023-03-26 11:55:28,367 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/0bdaa12f97f541edbf2a9d49a31b9409 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/0bdaa12f97f541edbf2a9d49a31b9409 2023-03-26 11:55:28,368 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/12880238fab34d2786accc4d5ba79039 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/12880238fab34d2786accc4d5ba79039 2023-03-26 11:55:28,369 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/78792a9bd2c549898e3ff27d7bfa7fd2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/78792a9bd2c549898e3ff27d7bfa7fd2 2023-03-26 11:55:28,371 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/ce158224643b497f8b96732ad86b16a1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/22ba5aaf9ccd49c6860967cafeeea388, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/98aefd7e9bd847cb9fbc432d7d09b2da, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/417818068c224b42b927ec7ff4b40c94, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/06356a2cbbae4d34b7224e45f1be0389, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/bbb7a7ce368c40e8878c6a1342268352, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/f9e490e75776447db4352bda688a42bc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/80efc585c52044f69c3325bcfd5912a7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1e5cb0d8c148433f8de9b459375f7ba2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/41268a48507041d18b2b1c2dbae185cb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/346c984b125e442c995c67c56b27ebe3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1afe82d74fcb4d4bb7345ccf7c9df990] to archive 2023-03-26 11:55:28,371 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:28,372 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/ce158224643b497f8b96732ad86b16a1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/ce158224643b497f8b96732ad86b16a1 2023-03-26 11:55:28,373 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/22ba5aaf9ccd49c6860967cafeeea388 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/22ba5aaf9ccd49c6860967cafeeea388 2023-03-26 11:55:28,374 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/98aefd7e9bd847cb9fbc432d7d09b2da to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/98aefd7e9bd847cb9fbc432d7d09b2da 2023-03-26 11:55:28,375 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/417818068c224b42b927ec7ff4b40c94 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/417818068c224b42b927ec7ff4b40c94 2023-03-26 11:55:28,375 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/06356a2cbbae4d34b7224e45f1be0389 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/06356a2cbbae4d34b7224e45f1be0389 2023-03-26 11:55:28,376 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/bbb7a7ce368c40e8878c6a1342268352 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/bbb7a7ce368c40e8878c6a1342268352 2023-03-26 11:55:28,377 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/f9e490e75776447db4352bda688a42bc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/f9e490e75776447db4352bda688a42bc 2023-03-26 11:55:28,377 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/80efc585c52044f69c3325bcfd5912a7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/80efc585c52044f69c3325bcfd5912a7 2023-03-26 11:55:28,378 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1e5cb0d8c148433f8de9b459375f7ba2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1e5cb0d8c148433f8de9b459375f7ba2 2023-03-26 11:55:28,379 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/41268a48507041d18b2b1c2dbae185cb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/41268a48507041d18b2b1c2dbae185cb 2023-03-26 11:55:28,379 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/346c984b125e442c995c67c56b27ebe3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/346c984b125e442c995c67c56b27ebe3 2023-03-26 11:55:28,380 DEBUG [StoreCloser-TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1afe82d74fcb4d4bb7345ccf7c9df990 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/1afe82d74fcb4d4bb7345ccf7c9df990 2023-03-26 11:55:28,383 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/recovered.edits/339.seqid, newMaxSeqId=339, maxSeqId=4 2023-03-26 11:55:28,384 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057. 2023-03-26 11:55:28,384 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for b57c4d4a48d50ba171b46d52c705a057: 2023-03-26 11:55:28,386 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:28,386 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=b57c4d4a48d50ba171b46d52c705a057, regionState=CLOSED 2023-03-26 11:55:28,386 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831728386"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831728386"}]},"ts":"1679831728386"} 2023-03-26 11:55:28,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-03-26 11:55:28,388 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure b57c4d4a48d50ba171b46d52c705a057, server=jenkins-hbase4.apache.org,38971,1679831615531 in 2.0340 sec 2023-03-26 11:55:28,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-03-26 11:55:28,389 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=b57c4d4a48d50ba171b46d52c705a057, UNASSIGN in 2.0380 sec 2023-03-26 11:55:28,389 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831728389"}]},"ts":"1679831728389"} 2023-03-26 11:55:28,390 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 11:55:28,391 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 11:55:28,393 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 2.0490 sec 2023-03-26 11:55:28,451 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-03-26 11:55:28,451 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-03-26 11:55:28,451 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 11:55:28,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,453 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,454 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-26 11:55:28,458 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:28,460 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/recovered.edits] 2023-03-26 11:55:28,462 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/20cba0ac4e924b55a0b16634388c1bc8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/20cba0ac4e924b55a0b16634388c1bc8 2023-03-26 11:55:28,463 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/50c77aa1c09144b58a644b0d8abe1ec7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/50c77aa1c09144b58a644b0d8abe1ec7 2023-03-26 11:55:28,464 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5a42fe72615b41da95cfce3b6b586ed3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/5a42fe72615b41da95cfce3b6b586ed3 2023-03-26 11:55:28,465 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9385c626589e49a6a9c8d1ba1c55b1a5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9385c626589e49a6a9c8d1ba1c55b1a5 2023-03-26 11:55:28,466 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9af3f4c125064950ad5b3d260e9dea01 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9af3f4c125064950ad5b3d260e9dea01 2023-03-26 11:55:28,466 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9cd69a2f14ef4133b0ed3d9725d008cf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/9cd69a2f14ef4133b0ed3d9725d008cf 2023-03-26 11:55:28,467 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/a525aff123954ecd9bfe19dc766bab96 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/a525aff123954ecd9bfe19dc766bab96 2023-03-26 11:55:28,468 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/b9d742d0dfe54a76a3c763fe531cdb6b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/A/b9d742d0dfe54a76a3c763fe531cdb6b 2023-03-26 11:55:28,470 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/21610e6106f04401a498659bb49b4b83 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/21610e6106f04401a498659bb49b4b83 2023-03-26 11:55:28,471 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/2d249229b25b4e199f746aa0897fc1dc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/2d249229b25b4e199f746aa0897fc1dc 2023-03-26 11:55:28,472 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/461b4bf845a24569b5f21767469b7580 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/461b4bf845a24569b5f21767469b7580 2023-03-26 11:55:28,473 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/56aecdb8ca7d4e6e9042ba88c987c550 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/56aecdb8ca7d4e6e9042ba88c987c550 2023-03-26 11:55:28,474 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/59601ca254c447c1986fbeaed4a960f5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/59601ca254c447c1986fbeaed4a960f5 2023-03-26 11:55:28,475 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9461b833afa34706bc7aa4df18b15f20 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/9461b833afa34706bc7aa4df18b15f20 2023-03-26 11:55:28,476 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/cf4c7cccb06e4be89d53fd81a6538120 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/cf4c7cccb06e4be89d53fd81a6538120 2023-03-26 11:55:28,477 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/f93a1d0a236e455792d8f576a3e11f93 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/B/f93a1d0a236e455792d8f576a3e11f93 2023-03-26 11:55:28,479 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/148d6bbe60be4d74801d1536f61f8571 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/148d6bbe60be4d74801d1536f61f8571 2023-03-26 11:55:28,480 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/16146a51686244618e9b87f88e56a76f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/16146a51686244618e9b87f88e56a76f 2023-03-26 11:55:28,480 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/38631826874b4130a05e3f80f53c5782 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/38631826874b4130a05e3f80f53c5782 2023-03-26 11:55:28,481 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/66bbaf7c4fc04fa6b5b617b2dff463a3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/66bbaf7c4fc04fa6b5b617b2dff463a3 2023-03-26 11:55:28,482 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/822d3d378a6643358d07fe89f8629579 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/822d3d378a6643358d07fe89f8629579 2023-03-26 11:55:28,483 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/82bf9813275d4957a585baa4fa4f3f66 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/82bf9813275d4957a585baa4fa4f3f66 2023-03-26 11:55:28,484 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/860abb228d9d472e9f11955b5399f9e4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/860abb228d9d472e9f11955b5399f9e4 2023-03-26 11:55:28,485 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/db95b3a3022c4a25847f549a880de465 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/C/db95b3a3022c4a25847f549a880de465 2023-03-26 11:55:28,486 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/recovered.edits/339.seqid to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057/recovered.edits/339.seqid 2023-03-26 11:55:28,487 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/b57c4d4a48d50ba171b46d52c705a057 2023-03-26 11:55:28,487 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:55:28,487 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 11:55:28,488 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-26 11:55:28,490 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032600eab2b4497546ebb79aa5afb12a7cc3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032600eab2b4497546ebb79aa5afb12a7cc3 2023-03-26 11:55:28,490 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326159d395f85ba4505b7aeb78624772538 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326159d395f85ba4505b7aeb78624772538 2023-03-26 11:55:28,491 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262a2b2648d62a4f588cc6b2513df384cb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303262a2b2648d62a4f588cc6b2513df384cb 2023-03-26 11:55:28,492 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032636c61fe18b014b5fbf52bbf6b39440f4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032636c61fe18b014b5fbf52bbf6b39440f4 2023-03-26 11:55:28,493 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263944f8f099734a4e9fe6c2b176ae0790 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263944f8f099734a4e9fe6c2b176ae0790 2023-03-26 11:55:28,494 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264752c8229bfb4270b19036726ee32528 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264752c8229bfb4270b19036726ee32528 2023-03-26 11:55:28,495 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032650f07478b80a4b95aa1f4b4bb57ffa54 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032650f07478b80a4b95aa1f4b4bb57ffa54 2023-03-26 11:55:28,495 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303266335f3c68dad44f2829f40ff9020b7e7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303266335f3c68dad44f2829f40ff9020b7e7 2023-03-26 11:55:28,496 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326655442da5d354a59a26be30ae8589959 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326655442da5d354a59a26be30ae8589959 2023-03-26 11:55:28,497 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267b5d6991d5e54b439d1f3ed0d69a94ed to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267b5d6991d5e54b439d1f3ed0d69a94ed 2023-03-26 11:55:28,498 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267f358ed7de294970b5629f1a038e273c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267f358ed7de294970b5629f1a038e273c 2023-03-26 11:55:28,499 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032699cef0f9f60345ebab961d8b45d63c83 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032699cef0f9f60345ebab961d8b45d63c83 2023-03-26 11:55:28,499 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b035c7dbf76e45bd9d40ae23a174a900 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b035c7dbf76e45bd9d40ae23a174a900 2023-03-26 11:55:28,500 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ba36184b28a14759a292a66177bc5ed7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ba36184b28a14759a292a66177bc5ed7 2023-03-26 11:55:28,501 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c8a24104d11645c3b41f3c1e40fb7467 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c8a24104d11645c3b41f3c1e40fb7467 2023-03-26 11:55:28,502 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326dd5b3051bbb940299e5916976b844cb4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326dd5b3051bbb940299e5916976b844cb4 2023-03-26 11:55:28,503 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e29a0cc6e03f404bb2b6961cb837c9c5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e29a0cc6e03f404bb2b6961cb837c9c5 2023-03-26 11:55:28,504 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2306281ea8b4b61b151bcdceab63674 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f2306281ea8b4b61b151bcdceab63674 2023-03-26 11:55:28,504 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 11:55:28,506 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,507 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 11:55:28,509 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 11:55:28,510 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,510 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 11:55:28,510 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679831728510"}]},"ts":"9223372036854775807"} 2023-03-26 11:55:28,511 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 11:55:28,511 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => b57c4d4a48d50ba171b46d52c705a057, NAME => 'TestAcidGuarantees,,1679831701416.b57c4d4a48d50ba171b46d52c705a057.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 11:55:28,511 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 11:55:28,511 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679831728511"}]},"ts":"9223372036854775807"} 2023-03-26 11:55:28,513 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 11:55:28,515 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,516 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 63 msec 2023-03-26 11:55:28,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-03-26 11:55:28,555 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-03-26 11:55:28,566 INFO [Listener at localhost/46409] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=228 (was 228), OpenFileDescriptor=540 (was 569), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=346 (was 366), ProcessCount=169 (was 169), AvailableMemoryMB=8730 (was 8831) 2023-03-26 11:55:28,577 INFO [Listener at localhost/46409] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=228, OpenFileDescriptor=540, MaxFileDescriptor=60000, SystemLoadAverage=346, ProcessCount=169, AvailableMemoryMB=8729 2023-03-26 11:55:28,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:55:28,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:55:28,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:28,581 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:55:28,581 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-03-26 11:55:28,582 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:55:28,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 11:55:28,583 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:28,583 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 empty. 2023-03-26 11:55:28,584 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:28,584 DEBUG [PEWorker-3] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:55:28,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 11:55:28,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 11:55:28,991 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:55:28,993 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => e1f18589e0dd3700bbf1b6b539ec6e88, NAME => 'TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:55:29,184 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 11:55:29,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:55:29,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing e1f18589e0dd3700bbf1b6b539ec6e88, disabling compactions & flushes 2023-03-26 11:55:29,406 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. after waiting 0 ms 2023-03-26 11:55:29,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,406 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,406 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:29,408 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:55:29,409 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831729409"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831729409"}]},"ts":"1679831729409"} 2023-03-26 11:55:29,410 INFO [PEWorker-3] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:55:29,411 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:55:29,411 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831729411"}]},"ts":"1679831729411"} 2023-03-26 11:55:29,412 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 11:55:29,418 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, ASSIGN}] 2023-03-26 11:55:29,420 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, ASSIGN 2023-03-26 11:55:29,420 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:55:29,571 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=e1f18589e0dd3700bbf1b6b539ec6e88, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:29,572 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831729571"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831729571"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831729571"}]},"ts":"1679831729571"} 2023-03-26 11:55:29,573 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:55:29,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 11:55:29,727 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,727 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => e1f18589e0dd3700bbf1b6b539ec6e88, NAME => 'TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:55:29,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:55:29,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,728 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,729 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,730 DEBUG [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A 2023-03-26 11:55:29,730 DEBUG [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A 2023-03-26 11:55:29,730 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:29,731 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1f18589e0dd3700bbf1b6b539ec6e88 columnFamilyName A 2023-03-26 11:55:29,731 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.HStore(310): Store=e1f18589e0dd3700bbf1b6b539ec6e88/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:29,731 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,732 DEBUG [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B 2023-03-26 11:55:29,732 DEBUG [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B 2023-03-26 11:55:29,733 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:29,733 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1f18589e0dd3700bbf1b6b539ec6e88 columnFamilyName B 2023-03-26 11:55:29,733 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.HStore(310): Store=e1f18589e0dd3700bbf1b6b539ec6e88/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:29,733 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,734 DEBUG [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C 2023-03-26 11:55:29,734 DEBUG [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C 2023-03-26 11:55:29,734 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:55:29,734 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region e1f18589e0dd3700bbf1b6b539ec6e88 columnFamilyName C 2023-03-26 11:55:29,735 INFO [StoreOpener-e1f18589e0dd3700bbf1b6b539ec6e88-1] regionserver.HStore(310): Store=e1f18589e0dd3700bbf1b6b539ec6e88/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:55:29,735 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,736 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,738 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:55:29,739 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:29,740 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:55:29,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened e1f18589e0dd3700bbf1b6b539ec6e88; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=75331380, jitterRate=0.12252503633499146}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:55:29,741 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:29,741 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., pid=49, masterSystemTime=1679831729724 2023-03-26 11:55:29,742 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,742 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:29,743 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=e1f18589e0dd3700bbf1b6b539ec6e88, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:29,743 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831729743"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831729743"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831729743"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831729743"}]},"ts":"1679831729743"} 2023-03-26 11:55:29,744 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-03-26 11:55:29,745 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 in 171 msec 2023-03-26 11:55:29,745 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-03-26 11:55:29,746 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, ASSIGN in 327 msec 2023-03-26 11:55:29,746 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:55:29,746 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831729746"}]},"ts":"1679831729746"} 2023-03-26 11:55:29,747 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 11:55:29,750 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:55:29,751 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-03-26 11:55:30,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-03-26 11:55:30,686 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-03-26 11:55:30,686 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x5ec943a9 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,691 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@27f2d997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,692 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,694 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46628, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,695 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:55:30,696 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57542, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:55:30,698 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x20ed9117 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,703 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34ad4aa8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,703 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x50002343 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,714 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1c29757d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,714 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x5726fb8b to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,720 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6edd017, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,720 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6f2b3c03 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,724 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@fb168ce, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,725 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x70e2be04 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,729 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@461f5f3f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,729 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x63093e21 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,734 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f5363d8, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,735 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x0b475d8d to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,738 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@211582f2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,738 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x549bef45 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,742 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@65816b6e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,743 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed183a4 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,747 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@653bb8fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,747 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x10a344e6 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:55:30,751 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@720d29f5, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:55:30,754 DEBUG [hconnection-0x483fead6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,755 DEBUG [hconnection-0x2a39f7cf-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,754 DEBUG [hconnection-0x7a783d03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,755 DEBUG [hconnection-0x7b86960b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,757 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46638, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,757 DEBUG [hconnection-0xc03c8c3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,757 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46634, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,757 DEBUG [hconnection-0x7be3b80e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,758 DEBUG [hconnection-0x4c0db68d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,759 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46648, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,759 DEBUG [hconnection-0x71963a55-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,759 DEBUG [hconnection-0x65ada4b8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,762 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46660, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,762 DEBUG [hconnection-0x33cc114a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:55:30,769 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46682, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,769 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:30,770 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46694, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,777 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46674, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,777 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46702, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,778 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46700, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,778 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46708, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:55:30,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:30,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:30,779 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:30,779 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:30,779 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:30,779 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:30,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:30,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,783 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:30,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:30,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:30,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:30,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:30,787 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 11:55:30,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:30,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:30,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:30,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:30,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:30,788 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:30,788 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:30,788 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:30,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:30,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:30,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:30,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:30,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:30,793 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27bcee02[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:30,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:30,795 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,796 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:30,796 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:30,796 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:30,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/ace8516e224149a983d225cff0ad0277 2023-03-26 11:55:30,798 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,798 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:30,798 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:30,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:30,798 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:30,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:30,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:30,801 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:30,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:30,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:30,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:30,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:30,801 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:30,803 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,803 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:30,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,803 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:30,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:30,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:30,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,805 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:30,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831790799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831790800, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831790801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:30,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:30,810 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:30,810 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:30,810 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,810 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:30,810 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,810 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:30,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:30,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831790807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,813 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831790808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,910 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:30,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:30,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:30,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831790909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831790909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,914 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831790909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,915 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831790912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:30,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831790914, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:30,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:30,919 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:30,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:30,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:30,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:30,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,920 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:30,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:30,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:30,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,921 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:30,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,922 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:30,922 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:30,922 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:30,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:30,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:30,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:30,924 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:30,924 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:30,924 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:30,924 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:30,924 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:30,931 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:30,931 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:30,931 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:30,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:30,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34a8726d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:30,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:30,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,934 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:30,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:30,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:30,935 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,935 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:30,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:30,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:30,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:30,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:30,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:30,939 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:30,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:30,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:30,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:30,939 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:30,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:30,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:30,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:30,941 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:30,943 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:30,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:30,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:30,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:30,946 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:30,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:30,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:30,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:30,947 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:30,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:30,947 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:30,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:30,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:31,047 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:31,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:31,049 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:31,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:31,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:31,052 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:31,052 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:31,052 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:31,052 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:31,052 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,052 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,054 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:31,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:31,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:31,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,054 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:31,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,055 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:31,056 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:31,056 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:31,056 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:31,056 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:31,056 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:31,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:31,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:31,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:31,060 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:31,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:31,065 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:31,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:31,066 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,066 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:31,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:31,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@176b728e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:31,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:31,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,068 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:31,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,068 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:31,068 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:31,069 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:31,070 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,070 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:31,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:31,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:31,072 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:31,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:31,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:31,072 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:31,072 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:31,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,074 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:31,075 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:31,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:31,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,076 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:31,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:31,077 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:31,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:31,079 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,079 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:31,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:31,079 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:31,079 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:31,079 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,080 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:31,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:31,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831791115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831791115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831791115, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,117 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831791116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,121 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831791118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,180 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:31,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:31,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:31,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:31,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:31,183 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:31,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:31,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:31,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:31,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,184 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:31,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:31,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,185 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:31,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:31,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:31,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:31,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:31,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:31,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:31,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:31,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:31,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:31,187 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:31,187 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:31,187 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:31,187 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:31,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:31,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:31,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:31,187 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:31,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:31,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:31,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:31,190 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7521f596[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:31,190 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:31,190 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,192 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:31,193 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:31,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:31,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:31,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:31,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:31,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:31,194 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:31,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:31,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:31,194 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:31,194 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:31,196 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,196 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,196 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:31,196 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:31,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:31,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:31,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:31,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:31,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:31,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:31,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:31,200 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:31,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:31,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,201 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,201 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,201 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:31,201 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:31,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:31,201 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,201 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:31,201 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:31,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:31,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:31,224 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/83bba8e92fc14272a84983ced8b0ef04 2023-03-26 11:55:31,242 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/efa3b5e0031d432abf7c00df04a180a4 2023-03-26 11:55:31,246 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/ace8516e224149a983d225cff0ad0277 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ace8516e224149a983d225cff0ad0277 2023-03-26 11:55:31,249 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ace8516e224149a983d225cff0ad0277, entries=200, sequenceid=16, filesize=13.9 K 2023-03-26 11:55:31,249 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/83bba8e92fc14272a84983ced8b0ef04 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/83bba8e92fc14272a84983ced8b0ef04 2023-03-26 11:55:31,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/83bba8e92fc14272a84983ced8b0ef04, entries=150, sequenceid=16, filesize=11.6 K 2023-03-26 11:55:31,254 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/efa3b5e0031d432abf7c00df04a180a4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/efa3b5e0031d432abf7c00df04a180a4 2023-03-26 11:55:31,257 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/efa3b5e0031d432abf7c00df04a180a4, entries=150, sequenceid=16, filesize=11.6 K 2023-03-26 11:55:31,258 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e1f18589e0dd3700bbf1b6b539ec6e88 in 470ms, sequenceid=16, compaction requested=false 2023-03-26 11:55:31,258 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:31,301 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:31,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:31,303 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:31,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:31,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:31,304 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:31,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:31,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:31,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:31,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,306 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,306 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:31,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:31,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,307 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:31,308 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:31,308 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:31,308 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:31,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:31,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:31,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:31,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:31,308 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:31,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:31,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:31,308 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 11:55:31,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:31,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:31,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:31,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:31,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:31,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:31,421 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831791428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831791429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,436 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831791432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831791433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,445 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831791436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831791537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831791537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,541 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831791537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,544 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831791541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,549 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831791546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,716 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/386913b4ea77405189cc692405c15778 2023-03-26 11:55:31,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831791742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831791742, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831791743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831791746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:31,754 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:31,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831791751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831792045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831792045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831792046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831792051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,058 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831792055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,126 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/46bcd1d220fb426899feace1c230f4af 2023-03-26 11:55:32,535 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/3c18647b7eee4faa9a9155cf16b4aa91 2023-03-26 11:55:32,539 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/386913b4ea77405189cc692405c15778 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/386913b4ea77405189cc692405c15778 2023-03-26 11:55:32,542 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/386913b4ea77405189cc692405c15778, entries=150, sequenceid=33, filesize=11.6 K 2023-03-26 11:55:32,543 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/46bcd1d220fb426899feace1c230f4af as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/46bcd1d220fb426899feace1c230f4af 2023-03-26 11:55:32,546 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/46bcd1d220fb426899feace1c230f4af, entries=150, sequenceid=33, filesize=11.6 K 2023-03-26 11:55:32,547 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/3c18647b7eee4faa9a9155cf16b4aa91 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3c18647b7eee4faa9a9155cf16b4aa91 2023-03-26 11:55:32,550 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3c18647b7eee4faa9a9155cf16b4aa91, entries=150, sequenceid=33, filesize=11.6 K 2023-03-26 11:55:32,550 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1242ms, sequenceid=33, compaction requested=false 2023-03-26 11:55:32,550 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:32,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:32,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:32,552 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,552 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,553 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,553 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:32,554 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:32,555 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35add9ce[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:32,555 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:32,555 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:32,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:32,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:32,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:32,555 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:32,555 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:32,557 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,557 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:32,557 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:32,557 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:32,558 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,558 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:32,559 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,559 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:32,559 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,559 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:32,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:32,562 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:32,562 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/771387e9a6f8441ead02f75f3228d44b 2023-03-26 11:55:32,562 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:32,562 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:32,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:32,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:32,563 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:32,564 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,565 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,565 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,566 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,566 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:32,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:32,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,569 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:32,569 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,569 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831792569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831792571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831792576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831792577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831792578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,669 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:32,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:32,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:32,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:32,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:32,673 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,673 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,675 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:32,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:32,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:32,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:32,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:32,677 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,677 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:32,677 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:32,677 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:32,677 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:32,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,679 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:32,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4261ee0c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:32,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,680 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:32,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831792679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,680 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:32,682 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:32,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,682 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:32,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831792681, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831792684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,686 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831792684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,686 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:32,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,686 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,687 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:32,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831792684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:32,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:32,689 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:32,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:32,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:32,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:32,689 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:32,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,691 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:32,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:32,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,695 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:32,695 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:32,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:32,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,795 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:32,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:32,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:32,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:32,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:32,798 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:32,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:32,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:32,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:32,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:32,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:32,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:32,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:32,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:32,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:32,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:32,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:32,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:32,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:32,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:32,809 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:32,813 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,813 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,814 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:32,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e521bcc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:32,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:32,815 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,816 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:32,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:32,816 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:32,816 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,816 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:32,817 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:32,817 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,817 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:32,820 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:32,823 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:32,823 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:32,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:32,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:32,823 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:32,823 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:32,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,824 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,825 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,832 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:32,832 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:32,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,832 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,832 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:32,832 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:32,832 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,884 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831792881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831792887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831792888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831792888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:32,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831792889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,932 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:32,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:32,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:32,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:32,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:32,948 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:32,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:32,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:32,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:32,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,950 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,950 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:32,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,952 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:32,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,953 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:32,963 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:32,963 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:32,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:32,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:32,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:32,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:32,965 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:32,965 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:32,967 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,967 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,968 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7425c2d5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:32,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:32,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,974 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/0c879baa896b4d3383d1e6c151f856f0 2023-03-26 11:55:32,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:32,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:32,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:32,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,977 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:32,977 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:32,981 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:32,981 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:32,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:32,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:32,981 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:32,981 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:32,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:32,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:32,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:32,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:32,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:32,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:32,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:32,988 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:32,988 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:32,988 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:32,988 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:32,988 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:32,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:32,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:32,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:32,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,088 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:33,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:33,090 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:33,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:33,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:33,091 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:33,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:33,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:33,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,095 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:33,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:33,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:33,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:33,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:33,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:33,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:33,097 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:33,097 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,098 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:33,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f9d0636[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:33,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:33,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,101 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:33,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:33,103 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:33,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:33,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:33,105 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:33,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:33,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:33,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:33,105 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:33,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:33,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,112 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:33,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:33,115 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:33,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:33,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:33,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,115 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:33,115 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,115 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831793185, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831793193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831793194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831793195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,199 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831793195, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,215 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:33,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:33,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:33,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:33,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:33,218 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:33,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:33,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:33,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,219 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,221 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,221 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,221 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:33,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,221 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:33,222 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:33,222 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:33,222 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,222 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:33,222 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:33,222 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:33,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:33,224 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:33,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62c18c47[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:33,225 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:33,225 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:33,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:33,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:33,229 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:33,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:33,232 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:33,232 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:33,232 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:33,232 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:33,232 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:33,233 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,233 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,233 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:33,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,234 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,237 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,239 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,239 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,239 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,239 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,239 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:33,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,240 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,240 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,240 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:33,240 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,240 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:33,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:33,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:33,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:33,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,240 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:33,240 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:33,340 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:33,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:33,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:33,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:33,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:33,343 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:33,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:33,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:33,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,345 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,345 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,345 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,345 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:33,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,346 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:33,346 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:33,346 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:33,347 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,347 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:33,347 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:33,347 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:33,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:33,349 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:33,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c5367a8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:33,351 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:33,351 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,352 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:33,352 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,352 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:33,352 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:33,353 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,353 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,354 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:33,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:33,356 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:33,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:33,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:33,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:33,356 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:33,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:33,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:33,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:33,358 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:33,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:33,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:33,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:33,362 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:33,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,363 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:33,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,363 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,363 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:33,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:33,363 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:33,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:33,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/c7b28a40807a436780bb992bcc9b7c8d 2023-03-26 11:55:33,406 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/771387e9a6f8441ead02f75f3228d44b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/771387e9a6f8441ead02f75f3228d44b 2023-03-26 11:55:33,410 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/771387e9a6f8441ead02f75f3228d44b, entries=250, sequenceid=50, filesize=16.2 K 2023-03-26 11:55:33,410 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/0c879baa896b4d3383d1e6c151f856f0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/0c879baa896b4d3383d1e6c151f856f0 2023-03-26 11:55:33,414 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/0c879baa896b4d3383d1e6c151f856f0, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 11:55:33,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/c7b28a40807a436780bb992bcc9b7c8d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/c7b28a40807a436780bb992bcc9b7c8d 2023-03-26 11:55:33,418 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/c7b28a40807a436780bb992bcc9b7c8d, entries=150, sequenceid=50, filesize=11.6 K 2023-03-26 11:55:33,419 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for e1f18589e0dd3700bbf1b6b539ec6e88 in 865ms, sequenceid=50, compaction requested=true 2023-03-26 11:55:33,419 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:33,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:33,419 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:33,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:33,419 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:33,419 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:33,420 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42678 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:33,420 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:33,420 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,421 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ace8516e224149a983d225cff0ad0277, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/386913b4ea77405189cc692405c15778, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/771387e9a6f8441ead02f75f3228d44b] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=41.7 K 2023-03-26 11:55:33,421 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:33,421 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:33,421 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,421 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/83bba8e92fc14272a84983ced8b0ef04, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/46bcd1d220fb426899feace1c230f4af, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/0c879baa896b4d3383d1e6c151f856f0] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=34.8 K 2023-03-26 11:55:33,421 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ace8516e224149a983d225cff0ad0277, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679831730769 2023-03-26 11:55:33,421 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 83bba8e92fc14272a84983ced8b0ef04, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679831730769 2023-03-26 11:55:33,422 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 386913b4ea77405189cc692405c15778, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679831730796 2023-03-26 11:55:33,422 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 46bcd1d220fb426899feace1c230f4af, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679831730796 2023-03-26 11:55:33,422 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 771387e9a6f8441ead02f75f3228d44b, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831731431 2023-03-26 11:55:33,422 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 0c879baa896b4d3383d1e6c151f856f0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831731432 2023-03-26 11:55:33,428 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#404 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:33,429 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#405 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:33,454 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/dbf98877c35b463d9bfc3cc2f1dc54ae as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dbf98877c35b463d9bfc3cc2f1dc54ae 2023-03-26 11:55:33,459 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into dbf98877c35b463d9bfc3cc2f1dc54ae(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:33,459 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:33,459 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831733419; duration=0sec 2023-03-26 11:55:33,459 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:33,459 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:33,461 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:33,461 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:33,462 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,462 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/efa3b5e0031d432abf7c00df04a180a4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3c18647b7eee4faa9a9155cf16b4aa91, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/c7b28a40807a436780bb992bcc9b7c8d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=34.8 K 2023-03-26 11:55:33,462 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting efa3b5e0031d432abf7c00df04a180a4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679831730769 2023-03-26 11:55:33,462 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3c18647b7eee4faa9a9155cf16b4aa91, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1679831730796 2023-03-26 11:55:33,463 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c7b28a40807a436780bb992bcc9b7c8d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831731432 2023-03-26 11:55:33,463 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:33,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:33,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:33,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:33,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:33,466 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:33,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:33,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:33,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,468 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:33,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:33,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:33,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:33,469 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:33,469 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#406 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:33,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:33,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:33,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:33,470 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:33,470 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:33,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:33,703 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:33,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:33,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831793716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,724 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831793717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831793722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831793723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,729 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831793724, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831793825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831793825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,834 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831793830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831793830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:33,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831793830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:33,854 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/2c9c5bb8aa9b4839941827afa34fcf03 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/2c9c5bb8aa9b4839941827afa34fcf03 2023-03-26 11:55:33,857 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into 2c9c5bb8aa9b4839941827afa34fcf03(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:33,858 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:33,858 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831733419; duration=0sec 2023-03-26 11:55:33,858 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:33,877 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/a1effe7a1c924576ac90ad2916edac40 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/a1effe7a1c924576ac90ad2916edac40 2023-03-26 11:55:33,880 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into a1effe7a1c924576ac90ad2916edac40(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:33,880 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/4e9dfe849fd9421197ad8ebd47e229a1 2023-03-26 11:55:33,880 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:33,880 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831733419; duration=0sec 2023-03-26 11:55:33,880 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:33,911 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 11:55:34,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831794030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831794030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831794035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831794036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831794036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,288 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/a8446ac49c4e4e7da3c1625c2d5d506f 2023-03-26 11:55:34,297 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/b562406250274acea9c4ba4315cebb17 2023-03-26 11:55:34,301 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/4e9dfe849fd9421197ad8ebd47e229a1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/4e9dfe849fd9421197ad8ebd47e229a1 2023-03-26 11:55:34,304 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/4e9dfe849fd9421197ad8ebd47e229a1, entries=150, sequenceid=66, filesize=11.6 K 2023-03-26 11:55:34,305 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/a8446ac49c4e4e7da3c1625c2d5d506f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a8446ac49c4e4e7da3c1625c2d5d506f 2023-03-26 11:55:34,309 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a8446ac49c4e4e7da3c1625c2d5d506f, entries=150, sequenceid=66, filesize=11.6 K 2023-03-26 11:55:34,309 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/b562406250274acea9c4ba4315cebb17 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b562406250274acea9c4ba4315cebb17 2023-03-26 11:55:34,312 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b562406250274acea9c4ba4315cebb17, entries=150, sequenceid=66, filesize=11.6 K 2023-03-26 11:55:34,313 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for e1f18589e0dd3700bbf1b6b539ec6e88 in 843ms, sequenceid=66, compaction requested=false 2023-03-26 11:55:34,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:34,313 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:34,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:34,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,315 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:34,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:34,317 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263a219d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:34,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:34,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,318 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,318 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:34,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:34,318 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:34,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:34,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:34,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:34,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:34,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:34,330 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:34,330 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:34,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:34,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:34,331 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:34,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,332 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,332 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,334 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:34,338 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 11:55:34,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,339 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,340 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,339 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:34,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:34,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:34,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:34,339 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:34,339 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:34,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:34,339 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,340 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:34,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,340 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:34,340 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:34,340 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:34,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831794373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831794378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831794379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831794380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831794380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,440 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:34,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:34,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:34,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:34,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:34,443 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:34,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:34,443 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:34,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,444 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,445 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,445 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,445 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,446 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:34,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,446 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:34,446 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:34,447 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:34,447 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,447 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:34,447 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:34,447 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:34,447 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:34,449 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,449 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:34,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61d12fda[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:34,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:34,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,452 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:34,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:34,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:34,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:34,456 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,456 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:34,459 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:34,459 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:34,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:34,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:34,459 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:34,459 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:34,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,462 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,464 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:34,465 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:34,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,465 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:34,465 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:34,465 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,465 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831794481, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831794483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831794486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831794486, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831794487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,565 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:34,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:34,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:34,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:34,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:34,568 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:34,568 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,568 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:34,568 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:34,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,571 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:34,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,572 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:34,572 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:34,572 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:34,572 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:34,573 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:34,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,574 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:34,576 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f9c2413[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:34,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:34,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,578 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:34,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,578 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:34,580 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:34,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:34,582 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:34,582 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:34,582 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:34,582 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:34,582 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:34,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,585 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,586 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:34,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:34,588 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,588 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,589 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,589 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:34,589 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:34,589 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:34,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831794685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,689 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:34,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:34,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:34,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:34,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:34,692 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:34,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:34,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:34,693 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,693 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831794689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831794691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831794691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831794691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,694 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,695 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,695 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:34,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:34,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:34,695 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:34,695 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:34,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:34,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:34,696 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:34,696 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:34,697 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,697 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:34,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68956c6f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:34,701 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:34,701 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,706 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,706 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:34,706 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:34,706 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:34,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:34,707 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,707 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,707 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,707 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:34,710 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:34,710 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:34,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:34,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:34,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:34,710 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:34,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,711 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,712 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,713 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,715 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:34,715 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,715 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,715 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:34,715 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,747 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/bf88d9517363400fbcf6e2c9dcc19b81 2023-03-26 11:55:34,756 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/bb3b0684bfeb41488385b5e45cd7f8c7 2023-03-26 11:55:34,816 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:34,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:34,818 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:34,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:34,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:34,819 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:34,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:34,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:34,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,819 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,825 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,825 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:34,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:34,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:34,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:34,826 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,826 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:34,826 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:34,826 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,826 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:34,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:34,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:34,827 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:34,828 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,828 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,828 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,829 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:34,830 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd49e29[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:34,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:34,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,831 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,831 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:34,831 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:34,831 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:34,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,831 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:34,833 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:34,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,833 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:34,835 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:34,835 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:34,835 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:34,835 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:34,835 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:34,835 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:34,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,837 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,837 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,838 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,838 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,839 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,841 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,841 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,841 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,841 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:34,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:34,841 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,842 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:34,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,842 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:34,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,942 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:34,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:34,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:34,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:34,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:34,945 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:34,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:34,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:34,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,947 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:34,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:34,949 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:34,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:34,949 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:34,949 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:34,949 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:34,949 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:34,949 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:34,949 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,949 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:34,949 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:34,949 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:34,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:34,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:34,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:34,950 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:34,951 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,951 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,951 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:34,953 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7713cdf6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:34,953 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:34,953 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,954 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,955 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:34,955 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:34,955 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:34,955 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,955 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:34,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:34,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,959 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:34,960 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:34,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:34,964 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:34,964 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:34,964 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:34,964 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:34,964 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:34,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:34,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,966 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:34,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:34,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:34,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:34,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:34,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:34,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:34,979 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:34,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:34,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:34,979 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:34,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:34,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831794988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:34,999 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:34,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831794996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831794996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831794996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,000 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831794996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,080 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:35,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:35,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:35,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:35,083 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:35,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:35,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:35,083 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:35,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,085 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:35,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:35,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:35,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,085 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,086 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:35,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,086 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:35,087 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:35,087 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:35,087 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:35,087 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:35,088 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:35,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:35,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:35,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:35,088 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:35,089 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,089 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:35,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:35,090 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:35,090 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:35,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:35,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:35,091 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1941c3bb[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:35,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:35,091 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:35,093 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:35,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:35,095 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:35,095 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:35,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:35,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:35,098 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:35,098 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:35,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:35,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:35,098 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:35,098 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:35,101 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,101 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,101 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:35,101 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:35,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:35,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,102 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:35,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,104 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:35,104 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:35,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:35,104 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:35,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:35,105 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:35,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,105 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:35,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:35,105 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,105 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:35,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:35,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:35,164 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=87 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/13344386cd744ae89556cf248b84e7f4 2023-03-26 11:55:35,167 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/bf88d9517363400fbcf6e2c9dcc19b81 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bf88d9517363400fbcf6e2c9dcc19b81 2023-03-26 11:55:35,171 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bf88d9517363400fbcf6e2c9dcc19b81, entries=250, sequenceid=87, filesize=16.2 K 2023-03-26 11:55:35,171 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/bb3b0684bfeb41488385b5e45cd7f8c7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/bb3b0684bfeb41488385b5e45cd7f8c7 2023-03-26 11:55:35,174 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/bb3b0684bfeb41488385b5e45cd7f8c7, entries=150, sequenceid=87, filesize=11.6 K 2023-03-26 11:55:35,175 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/13344386cd744ae89556cf248b84e7f4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/13344386cd744ae89556cf248b84e7f4 2023-03-26 11:55:35,178 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/13344386cd744ae89556cf248b84e7f4, entries=150, sequenceid=87, filesize=11.6 K 2023-03-26 11:55:35,178 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for e1f18589e0dd3700bbf1b6b539ec6e88 in 841ms, sequenceid=87, compaction requested=true 2023-03-26 11:55:35,178 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:35,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:35,178 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:35,179 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:35,178 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:35,179 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:35,179 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 40441 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:35,179 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:35,179 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:35,180 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:35,180 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:35,180 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:35,180 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/2c9c5bb8aa9b4839941827afa34fcf03, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/4e9dfe849fd9421197ad8ebd47e229a1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bf88d9517363400fbcf6e2c9dcc19b81] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=39.5 K 2023-03-26 11:55:35,180 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dbf98877c35b463d9bfc3cc2f1dc54ae, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a8446ac49c4e4e7da3c1625c2d5d506f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/bb3b0684bfeb41488385b5e45cd7f8c7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=34.9 K 2023-03-26 11:55:35,180 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 2c9c5bb8aa9b4839941827afa34fcf03, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831731432 2023-03-26 11:55:35,180 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dbf98877c35b463d9bfc3cc2f1dc54ae, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831731432 2023-03-26 11:55:35,180 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 4e9dfe849fd9421197ad8ebd47e229a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679831732567 2023-03-26 11:55:35,180 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a8446ac49c4e4e7da3c1625c2d5d506f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679831732567 2023-03-26 11:55:35,181 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting bf88d9517363400fbcf6e2c9dcc19b81, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679831733710 2023-03-26 11:55:35,181 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bb3b0684bfeb41488385b5e45cd7f8c7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679831733721 2023-03-26 11:55:35,190 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#414 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:35,190 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#413 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:35,205 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/e1154ac03a7541f8bfb481175112ac57 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/e1154ac03a7541f8bfb481175112ac57 2023-03-26 11:55:35,205 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:35,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:35,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:35,209 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into e1154ac03a7541f8bfb481175112ac57(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:35,209 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:35,209 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831735178; duration=0sec 2023-03-26 11:55:35,209 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:35,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:35,210 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:35,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:35,210 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:35,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:35,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:35,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:35,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,210 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,210 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:35,211 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:35,211 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:35,211 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/a1effe7a1c924576ac90ad2916edac40, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b562406250274acea9c4ba4315cebb17, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/13344386cd744ae89556cf248b84e7f4] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=34.9 K 2023-03-26 11:55:35,211 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a1effe7a1c924576ac90ad2916edac40, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1679831731432 2023-03-26 11:55:35,211 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,211 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:35,211 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:35,211 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b562406250274acea9c4ba4315cebb17, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1679831732567 2023-03-26 11:55:35,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:35,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,212 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 13344386cd744ae89556cf248b84e7f4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679831733721 2023-03-26 11:55:35,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:35,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:35,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:35,213 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:35,213 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:35,213 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:35,213 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:35,213 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:35,214 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:35,214 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:35,214 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:35,217 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:35,231 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/b8e552cfd05a4cf98f7ec16a1caaebaf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b8e552cfd05a4cf98f7ec16a1caaebaf 2023-03-26 11:55:35,235 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into b8e552cfd05a4cf98f7ec16a1caaebaf(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:35,235 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:35,235 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831735179; duration=0sec 2023-03-26 11:55:35,235 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:35,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:35,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:35,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831795514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831795514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831795516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831795516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831795517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,606 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/383307fcb33a49538a088a5d3f3242dd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/383307fcb33a49538a088a5d3f3242dd 2023-03-26 11:55:35,610 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 383307fcb33a49538a088a5d3f3242dd(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:35,610 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:35,610 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831735178; duration=0sec 2023-03-26 11:55:35,610 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:35,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831795620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,624 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/586fe521d1e24649ba9b19cbf4f6bbf5 2023-03-26 11:55:35,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831795621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831795626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831795626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,630 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831795626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831795825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831795825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831795831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831795832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:35,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:35,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831795832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,034 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/696858f039cb43ba941f89685eaac36e 2023-03-26 11:55:36,131 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831796128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,133 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831796130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831796135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831796136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831796136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,457 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=102 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/670db2ed6f0d44e58e36204b2b8009bf 2023-03-26 11:55:36,461 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/586fe521d1e24649ba9b19cbf4f6bbf5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/586fe521d1e24649ba9b19cbf4f6bbf5 2023-03-26 11:55:36,464 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/586fe521d1e24649ba9b19cbf4f6bbf5, entries=150, sequenceid=102, filesize=11.6 K 2023-03-26 11:55:36,465 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/696858f039cb43ba941f89685eaac36e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/696858f039cb43ba941f89685eaac36e 2023-03-26 11:55:36,468 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/696858f039cb43ba941f89685eaac36e, entries=150, sequenceid=102, filesize=11.6 K 2023-03-26 11:55:36,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/670db2ed6f0d44e58e36204b2b8009bf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/670db2ed6f0d44e58e36204b2b8009bf 2023-03-26 11:55:36,473 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/670db2ed6f0d44e58e36204b2b8009bf, entries=150, sequenceid=102, filesize=11.6 K 2023-03-26 11:55:36,473 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1259ms, sequenceid=102, compaction requested=false 2023-03-26 11:55:36,474 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:36,474 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:36,474 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:36,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:36,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:36,480 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a7432e8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:36,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:36,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,482 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:36,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,482 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:36,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:36,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:36,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:36,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:36,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:36,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:36,486 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:36,486 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:36,486 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:36,486 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:36,486 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:36,487 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,487 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,488 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:36,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:36,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:36,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:36,489 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:36,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:36,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:36,493 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:36,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:36,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:36,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:36,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:36,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:36,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:36,494 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:36,494 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:36,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:36,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:36,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:36,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,594 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:36,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:36,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:36,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:36,597 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:36,597 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:36,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:36,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:36,597 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:36,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,598 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:36,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:36,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:36,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:36,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:36,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:36,600 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:36,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:36,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:36,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:36,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:36,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:36,601 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:36,601 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:36,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:36,635 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:36,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831796646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831796646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831796650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831796651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,657 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831796652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831796753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,757 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831796753, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831796758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831796759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,763 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831796759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,927 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-03-26 11:55:36,938 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ace8516e224149a983d225cff0ad0277, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/386913b4ea77405189cc692405c15778, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/771387e9a6f8441ead02f75f3228d44b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/2c9c5bb8aa9b4839941827afa34fcf03, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/4e9dfe849fd9421197ad8ebd47e229a1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bf88d9517363400fbcf6e2c9dcc19b81] to archive 2023-03-26 11:55:36,939 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:36,940 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ace8516e224149a983d225cff0ad0277 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ace8516e224149a983d225cff0ad0277 2023-03-26 11:55:36,941 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/386913b4ea77405189cc692405c15778 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/386913b4ea77405189cc692405c15778 2023-03-26 11:55:36,942 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/771387e9a6f8441ead02f75f3228d44b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/771387e9a6f8441ead02f75f3228d44b 2023-03-26 11:55:36,943 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/2c9c5bb8aa9b4839941827afa34fcf03 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/2c9c5bb8aa9b4839941827afa34fcf03 2023-03-26 11:55:36,944 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/4e9dfe849fd9421197ad8ebd47e229a1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/4e9dfe849fd9421197ad8ebd47e229a1 2023-03-26 11:55:36,945 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bf88d9517363400fbcf6e2c9dcc19b81 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bf88d9517363400fbcf6e2c9dcc19b81 2023-03-26 11:55:36,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/83bba8e92fc14272a84983ced8b0ef04, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/46bcd1d220fb426899feace1c230f4af, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dbf98877c35b463d9bfc3cc2f1dc54ae, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/0c879baa896b4d3383d1e6c151f856f0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a8446ac49c4e4e7da3c1625c2d5d506f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/bb3b0684bfeb41488385b5e45cd7f8c7] to archive 2023-03-26 11:55:36,957 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:36,959 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/83bba8e92fc14272a84983ced8b0ef04 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/83bba8e92fc14272a84983ced8b0ef04 2023-03-26 11:55:36,960 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/46bcd1d220fb426899feace1c230f4af to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/46bcd1d220fb426899feace1c230f4af 2023-03-26 11:55:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831796957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831796958, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,961 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dbf98877c35b463d9bfc3cc2f1dc54ae to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dbf98877c35b463d9bfc3cc2f1dc54ae 2023-03-26 11:55:36,963 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/0c879baa896b4d3383d1e6c151f856f0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/0c879baa896b4d3383d1e6c151f856f0 2023-03-26 11:55:36,964 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a8446ac49c4e4e7da3c1625c2d5d506f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a8446ac49c4e4e7da3c1625c2d5d506f 2023-03-26 11:55:36,965 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/bb3b0684bfeb41488385b5e45cd7f8c7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/bb3b0684bfeb41488385b5e45cd7f8c7 2023-03-26 11:55:36,972 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831796961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,977 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/efa3b5e0031d432abf7c00df04a180a4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3c18647b7eee4faa9a9155cf16b4aa91, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/a1effe7a1c924576ac90ad2916edac40, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/c7b28a40807a436780bb992bcc9b7c8d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b562406250274acea9c4ba4315cebb17, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/13344386cd744ae89556cf248b84e7f4] to archive 2023-03-26 11:55:36,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831796965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,978 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:36,978 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:36,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831796965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:36,979 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/efa3b5e0031d432abf7c00df04a180a4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/efa3b5e0031d432abf7c00df04a180a4 2023-03-26 11:55:36,980 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3c18647b7eee4faa9a9155cf16b4aa91 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3c18647b7eee4faa9a9155cf16b4aa91 2023-03-26 11:55:36,982 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/a1effe7a1c924576ac90ad2916edac40 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/a1effe7a1c924576ac90ad2916edac40 2023-03-26 11:55:36,983 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/c7b28a40807a436780bb992bcc9b7c8d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/c7b28a40807a436780bb992bcc9b7c8d 2023-03-26 11:55:36,984 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b562406250274acea9c4ba4315cebb17 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b562406250274acea9c4ba4315cebb17 2023-03-26 11:55:36,985 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/13344386cd744ae89556cf248b84e7f4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/13344386cd744ae89556cf248b84e7f4 2023-03-26 11:55:37,008 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/52669b68ffcf45548d312a328c97cde3 2023-03-26 11:55:37,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831797262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,267 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831797263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831797276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,280 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831797278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,286 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831797280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,421 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/e0b28e452bfa4fc89a5bf657296e8462 2023-03-26 11:55:37,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831797769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,775 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831797771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,786 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831797783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,792 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831797784, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,795 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:37,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831797789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,835 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=122 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/bb72f3be41b4440fb9d844f34e08fedf 2023-03-26 11:55:37,839 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/52669b68ffcf45548d312a328c97cde3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/52669b68ffcf45548d312a328c97cde3 2023-03-26 11:55:37,843 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/52669b68ffcf45548d312a328c97cde3, entries=150, sequenceid=122, filesize=11.6 K 2023-03-26 11:55:37,844 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/e0b28e452bfa4fc89a5bf657296e8462 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e0b28e452bfa4fc89a5bf657296e8462 2023-03-26 11:55:37,847 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e0b28e452bfa4fc89a5bf657296e8462, entries=150, sequenceid=122, filesize=11.6 K 2023-03-26 11:55:37,847 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/bb72f3be41b4440fb9d844f34e08fedf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/bb72f3be41b4440fb9d844f34e08fedf 2023-03-26 11:55:37,850 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/bb72f3be41b4440fb9d844f34e08fedf, entries=150, sequenceid=122, filesize=11.6 K 2023-03-26 11:55:37,851 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1250ms, sequenceid=122, compaction requested=true 2023-03-26 11:55:37,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:37,851 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:37,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:37,853 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:37,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:37,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:37,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:37,854 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@61eaca59[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:37,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:37,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:37,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:37,856 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:37,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:37,858 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:37,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:37,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:37,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:37,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:37,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:37,860 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:37,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:37,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:37,860 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:37,861 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:37,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:37,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:37,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:37,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:37,863 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:37,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:37,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:37,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:37,866 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:37,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:37,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:37,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:37,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:37,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,866 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:37,867 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:37,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:37,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:37,967 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:37,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:37,969 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:37,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:37,976 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:37,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:37,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:37,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:37,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:37,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:37,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:37,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:37,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:37,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:37,978 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:37,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:37,979 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:37,979 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:37,979 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:37,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:37,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:37,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:37,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:37,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:37,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:37,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:37,980 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 11:55:37,981 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:37,981 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:37,981 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:37,981 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:37,981 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:37,981 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:37,993 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/d025ccb09bb848ae9bfecff8f88fcd05 2023-03-26 11:55:38,009 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/f08c8c05347c411a9938db1b51ba4218 2023-03-26 11:55:38,430 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=136 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/4860150ca2b741979e8e3b45e1688129 2023-03-26 11:55:38,438 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/d025ccb09bb848ae9bfecff8f88fcd05 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/d025ccb09bb848ae9bfecff8f88fcd05 2023-03-26 11:55:38,442 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/d025ccb09bb848ae9bfecff8f88fcd05, entries=150, sequenceid=136, filesize=11.8 K 2023-03-26 11:55:38,443 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/f08c8c05347c411a9938db1b51ba4218 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/f08c8c05347c411a9938db1b51ba4218 2023-03-26 11:55:38,448 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/f08c8c05347c411a9938db1b51ba4218, entries=150, sequenceid=136, filesize=11.8 K 2023-03-26 11:55:38,449 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/4860150ca2b741979e8e3b45e1688129 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4860150ca2b741979e8e3b45e1688129 2023-03-26 11:55:38,453 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4860150ca2b741979e8e3b45e1688129, entries=150, sequenceid=136, filesize=11.8 K 2023-03-26 11:55:38,459 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=0 B/0 for e1f18589e0dd3700bbf1b6b539ec6e88 in 479ms, sequenceid=136, compaction requested=true 2023-03-26 11:55:38,459 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:38,459 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:38,459 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,462 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,462 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,463 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:38,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@87fd7ef[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:38,467 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:38,468 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:38,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:38,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:38,469 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:38,471 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:38,471 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,471 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,471 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:38,477 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:38,477 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:38,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:38,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:38,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:38,477 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:38,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,479 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,480 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,484 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:38,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,484 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,485 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:38,488 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:38,488 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:38,588 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:38,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:38,590 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:38,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:38,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:38,591 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:38,591 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,591 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:38,591 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:38,592 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,592 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,593 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,593 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:38,594 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:38,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:38,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:38,595 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:38,595 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,595 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:38,595 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:38,595 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:38,595 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,597 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,597 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:38,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:38,599 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fb49140[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:38,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:38,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:38,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:38,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:38,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,600 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:38,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:38,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:38,609 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:38,609 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:38,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:38,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:38,609 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:38,609 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:38,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,611 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,611 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,612 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,615 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,615 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,615 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:38,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:38,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,616 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:38,616 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:38,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,616 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:38,617 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,617 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,716 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:38,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:38,718 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:38,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:38,719 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:38,719 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:38,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:38,719 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:38,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,721 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,721 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,722 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,722 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:38,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:38,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:38,723 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:38,723 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:38,723 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,723 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:38,723 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:38,723 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:38,723 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:38,725 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:38,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:38,727 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ac132c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:38,727 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:38,727 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,728 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,728 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:38,728 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:38,728 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:38,729 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,729 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:38,730 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:38,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:38,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:38,733 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:38,733 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:38,733 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:38,733 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:38,733 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:38,734 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,735 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,735 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,736 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:38,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,739 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,739 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:38,740 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:38,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:38,792 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:55:38,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:38,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:38,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:38,792 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:38,792 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:38,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:38,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831738808, max timestamp=1679831738823Num uniques 859; 2023-03-26 11:55:38,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:55:38,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831738808, max timestamp=1679831738823Num uniques 859; 2023-03-26 11:55:38,825 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:55:38,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831738808, max timestamp=1679831738823Num uniques 859; 2023-03-26 11:55:38,826 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:55:38,840 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:38,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:38,842 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:38,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:38,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:38,844 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:38,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:38,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:38,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,844 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,845 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,846 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:38,847 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:38,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,847 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:38,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:38,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:38,848 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,848 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:38,848 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:38,848 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:38,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:38,849 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:38,851 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,851 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:38,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:38,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53a3a5e2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:38,853 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:38,853 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:38,855 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:38,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:38,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:38,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,857 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:38,858 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:38,859 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:38,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:38,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:38,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:38,859 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:38,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,861 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,861 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831798852, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,861 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,862 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831798853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831798854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:38,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:38,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:38,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,868 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,868 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,868 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,867 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:38,868 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:38,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:38,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:38,869 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:38,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831798860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831798861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,968 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:38,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:38,970 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:38,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:38,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:38,971 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:38,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:38,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:38,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,973 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,974 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:38,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:38,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,974 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:38,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,975 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:38,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:38,975 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:38,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,976 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:38,976 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:38,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831798962, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:38,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:38,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:38,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831798965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,976 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,976 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831798965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:38,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:38,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:38,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:38,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:38,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:38,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:38,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:38,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:38,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2595d0d6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:38,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:38,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:38,986 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:38,986 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:38,986 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:38,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831798974, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:38,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831798975, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:38,990 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:38,992 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:38,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:38,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:38,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:38,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:38,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:38,995 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:38,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:38,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:38,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:38,995 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:39,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,000 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,006 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:39,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:39,008 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:39,008 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:39,012 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:39,108 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:39,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:39,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:39,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:39,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:39,114 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:39,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:39,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:39,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,115 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,117 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,118 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,118 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:39,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:39,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:39,123 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:39,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:39,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:39,124 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,124 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:39,124 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:39,124 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:39,124 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:39,125 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,125 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,126 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,126 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:39,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e907bb5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:39,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:39,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:39,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:39,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:39,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:39,130 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,130 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,130 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,130 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:39,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:39,132 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:39,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:39,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:39,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:39,132 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:39,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,135 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:39,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,139 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:39,139 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:39,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,140 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,140 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831799178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831799178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,190 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,190 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831799178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,198 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831799188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,200 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/81a130a1a025410f98900698ed59ffe8 2023-03-26 11:55:39,205 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831799194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,240 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:39,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:39,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:39,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:39,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:39,242 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:39,242 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,242 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:39,242 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:39,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,244 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,244 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:39,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:39,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:39,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:39,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:39,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:39,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:39,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:39,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,251 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:39,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:39,253 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa13e38[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:39,253 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:39,253 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,255 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:39,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:39,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:39,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:39,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:39,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:39,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:39,260 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:39,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:39,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:39,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:39,260 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:39,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,262 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,264 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:39,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:39,267 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,267 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,267 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:39,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,268 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:39,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,368 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:39,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:39,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:39,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:39,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:39,371 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:39,371 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,371 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:39,371 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:39,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,374 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:39,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:39,376 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:39,376 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:39,376 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:39,376 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:39,378 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:39,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54e3043b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:39,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:39,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:39,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:39,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:39,381 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:39,382 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:39,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:39,385 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:39,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:39,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:39,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:39,385 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:39,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,400 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,400 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:39,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:39,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:39,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,401 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:39,401 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:39,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831799491, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831799493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,497 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831799493, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,501 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:39,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:39,503 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:39,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:39,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:39,504 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:39,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831799500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,508 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:39,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,508 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:39,509 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:39,509 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:39,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:39,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,509 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:39,511 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:39,511 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:39,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:39,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:39,511 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:39,512 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:39,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:39,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831799506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,519 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,520 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:39,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f7b6d11[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:39,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:39,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,527 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,528 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:39,528 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:39,528 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:39,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:39,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:39,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:39,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:39,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:39,535 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:39,535 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:39,535 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:39,535 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:39,535 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:39,537 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:39,537 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,537 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:39,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,538 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,539 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:39,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,542 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:39,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:39,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:39,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:39,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,542 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,543 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,543 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:39,543 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,543 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:39,543 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:39,543 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:39,543 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:39,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:39,609 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/6da68ac6b94b443abd2395c5c3fceae7 2023-03-26 11:55:39,625 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=148 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/6299424c2b0b4b37b91d63b79000a84c 2023-03-26 11:55:39,629 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/81a130a1a025410f98900698ed59ffe8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/81a130a1a025410f98900698ed59ffe8 2023-03-26 11:55:39,632 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/81a130a1a025410f98900698ed59ffe8, entries=250, sequenceid=148, filesize=16.4 K 2023-03-26 11:55:39,633 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/6da68ac6b94b443abd2395c5c3fceae7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6da68ac6b94b443abd2395c5c3fceae7 2023-03-26 11:55:39,637 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6da68ac6b94b443abd2395c5c3fceae7, entries=150, sequenceid=148, filesize=11.8 K 2023-03-26 11:55:39,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/6299424c2b0b4b37b91d63b79000a84c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6299424c2b0b4b37b91d63b79000a84c 2023-03-26 11:55:39,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6299424c2b0b4b37b91d63b79000a84c, entries=150, sequenceid=148, filesize=11.8 K 2023-03-26 11:55:39,643 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:39,643 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e1f18589e0dd3700bbf1b6b539ec6e88 in 851ms, sequenceid=148, compaction requested=true 2023-03-26 11:55:39,643 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:39,643 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:39,643 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:55:39,644 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:55:39,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:39,644 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:39,646 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:39,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:39,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:39,647 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59936 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:55:39,647 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:39,647 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,647 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/383307fcb33a49538a088a5d3f3242dd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/696858f039cb43ba941f89685eaac36e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e0b28e452bfa4fc89a5bf657296e8462, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/f08c8c05347c411a9938db1b51ba4218, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6da68ac6b94b443abd2395c5c3fceae7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=58.5 K 2023-03-26 11:55:39,647 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:39,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:39,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:39,647 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 64716 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:55:39,648 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:39,648 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,648 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/e1154ac03a7541f8bfb481175112ac57, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/586fe521d1e24649ba9b19cbf4f6bbf5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/52669b68ffcf45548d312a328c97cde3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/d025ccb09bb848ae9bfecff8f88fcd05, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/81a130a1a025410f98900698ed59ffe8] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=63.2 K 2023-03-26 11:55:39,648 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,648 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,648 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 383307fcb33a49538a088a5d3f3242dd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679831733721 2023-03-26 11:55:39,648 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 696858f039cb43ba941f89685eaac36e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679831734368 2023-03-26 11:55:39,648 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting e1154ac03a7541f8bfb481175112ac57, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679831733721 2023-03-26 11:55:39,649 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 586fe521d1e24649ba9b19cbf4f6bbf5, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679831734368 2023-03-26 11:55:39,649 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e0b28e452bfa4fc89a5bf657296e8462, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679831735513 2023-03-26 11:55:39,649 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:39,649 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:39,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:39,650 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f08c8c05347c411a9938db1b51ba4218, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679831736648 2023-03-26 11:55:39,650 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 52669b68ffcf45548d312a328c97cde3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679831735513 2023-03-26 11:55:39,650 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:39,650 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:39,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,650 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6da68ac6b94b443abd2395c5c3fceae7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679831738784 2023-03-26 11:55:39,650 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting d025ccb09bb848ae9bfecff8f88fcd05, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679831736648 2023-03-26 11:55:39,650 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:39,651 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 81a130a1a025410f98900698ed59ffe8, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679831738777 2023-03-26 11:55:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:39,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:39,651 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:39,651 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:39,651 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:39,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:39,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:39,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:39,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:39,651 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:39,651 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,651 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:39,652 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=134.18 KB heapSize=375.61 KB 2023-03-26 11:55:39,652 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:39,652 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:39,652 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:39,652 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:39,652 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:39,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:39,674 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#429 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:39,675 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#430 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:39,735 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/8e32cfcdae1c4882b5ee9079c17953f9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/8e32cfcdae1c4882b5ee9079c17953f9 2023-03-26 11:55:39,737 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/90617ece3b19443aaf3acf3f8a936aad as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/90617ece3b19443aaf3acf3f8a936aad 2023-03-26 11:55:39,740 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 8e32cfcdae1c4882b5ee9079c17953f9(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:39,740 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:39,740 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=11, startTime=1679831739644; duration=0sec 2023-03-26 11:55:39,740 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:39,740 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:55:39,742 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into 90617ece3b19443aaf3acf3f8a936aad(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:39,742 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:39,742 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=11, startTime=1679831739643; duration=0sec 2023-03-26 11:55:39,742 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59936 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:55:39,742 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:39,742 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:39,743 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:39,743 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b8e552cfd05a4cf98f7ec16a1caaebaf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/670db2ed6f0d44e58e36204b2b8009bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/bb72f3be41b4440fb9d844f34e08fedf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4860150ca2b741979e8e3b45e1688129, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6299424c2b0b4b37b91d63b79000a84c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=58.5 K 2023-03-26 11:55:39,743 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b8e552cfd05a4cf98f7ec16a1caaebaf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=87, earliestPutTs=1679831733721 2023-03-26 11:55:39,743 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 670db2ed6f0d44e58e36204b2b8009bf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=102, earliestPutTs=1679831734368 2023-03-26 11:55:39,744 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bb72f3be41b4440fb9d844f34e08fedf, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=122, earliestPutTs=1679831735513 2023-03-26 11:55:39,744 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 4860150ca2b741979e8e3b45e1688129, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=136, earliestPutTs=1679831736648 2023-03-26 11:55:39,744 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6299424c2b0b4b37b91d63b79000a84c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679831738784 2023-03-26 11:55:39,752 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#431 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:39,780 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/5223938ea0fb4953830ab7f9c7a3685b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5223938ea0fb4953830ab7f9c7a3685b 2023-03-26 11:55:39,791 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into 5223938ea0fb4953830ab7f9c7a3685b(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:39,792 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:39,792 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=11, startTime=1679831739644; duration=0sec 2023-03-26 11:55:39,793 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:40,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:40,004 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:40,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831800010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,017 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831800011, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,022 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831800015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,023 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831800017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,025 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831800017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,086 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/ec9b3d9f44e648d48c09ed3fc620b9bf 2023-03-26 11:55:40,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831800118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,128 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831800124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,128 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831800126, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,326 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831800323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831800329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,332 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,332 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831800329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,497 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/b11d00589bf0455b9210f6b5e4e22d0c 2023-03-26 11:55:40,632 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831800629, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831800633, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:40,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831800635, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,918 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=183 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/217bda0b520c477ea95ba358b25eb838 2023-03-26 11:55:40,922 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/ec9b3d9f44e648d48c09ed3fc620b9bf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ec9b3d9f44e648d48c09ed3fc620b9bf 2023-03-26 11:55:40,926 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ec9b3d9f44e648d48c09ed3fc620b9bf, entries=150, sequenceid=183, filesize=11.8 K 2023-03-26 11:55:40,926 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/b11d00589bf0455b9210f6b5e4e22d0c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/b11d00589bf0455b9210f6b5e4e22d0c 2023-03-26 11:55:40,930 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/b11d00589bf0455b9210f6b5e4e22d0c, entries=150, sequenceid=183, filesize=11.8 K 2023-03-26 11:55:40,930 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/217bda0b520c477ea95ba358b25eb838 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/217bda0b520c477ea95ba358b25eb838 2023-03-26 11:55:40,933 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/217bda0b520c477ea95ba358b25eb838, entries=150, sequenceid=183, filesize=11.8 K 2023-03-26 11:55:40,934 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~375.56 KB/384576, currentSize=46.96 KB/48090 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1282ms, sequenceid=183, compaction requested=false 2023-03-26 11:55:40,934 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:40,934 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:40,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:40,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:40,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:40,934 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:40,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:40,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,936 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:40,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:40,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42abf898[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:40,938 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:40,938 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,939 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:40,940 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:40,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:40,940 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:40,943 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,943 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:40,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:40,943 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:40,943 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:40,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:40,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:40,946 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:40,946 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:40,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:40,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:40,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:40,946 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:40,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,947 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:40,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:40,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:40,948 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:40,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:40,949 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:40,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:40,951 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:40,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:40,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:40,951 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:40,951 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:40,952 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:40,952 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:40,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:40,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:41,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:55:41,021 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:41,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:41,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:41,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:41,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:41,022 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:41,028 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/8f755ffa8091437aa2068586cfcabebb 2023-03-26 11:55:41,052 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:41,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:41,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:41,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:41,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:41,054 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:41,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:41,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:41,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,055 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:41,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,057 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,057 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:41,057 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:41,057 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,057 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:41,058 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,058 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:41,058 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:41,058 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:41,058 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:41,060 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,060 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:41,061 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51e84e0b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:41,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:41,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:41,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:41,063 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:41,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,064 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:41,065 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,065 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:41,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:41,067 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:41,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:41,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:41,067 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:41,067 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:41,068 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,068 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,068 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,069 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,070 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,077 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,077 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,078 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,077 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:41,078 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:41,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,078 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:41,078 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:41,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831741034, max timestamp=1679831741083Num uniques 859; 2023-03-26 11:55:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:55:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831741034, max timestamp=1679831741083Num uniques 859; 2023-03-26 11:55:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:55:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831741034, max timestamp=1679831741083Num uniques 859; 2023-03-26 11:55:41,085 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:55:41,141 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831801138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831801139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831801140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831801140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,152 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831801141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,178 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:41,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:41,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:41,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:41,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:41,181 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:41,181 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,182 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:41,182 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:41,182 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,182 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,186 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:41,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:41,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,187 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:41,188 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:41,188 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:41,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:41,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:41,192 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,192 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:41,192 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:41,192 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:41,192 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:41,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:41,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1de64ad1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:41,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:41,198 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,200 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:41,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:41,200 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:41,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:41,202 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:41,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,202 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:41,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:41,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:41,205 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:41,205 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:41,205 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:41,205 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:41,205 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:41,207 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,207 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,207 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,207 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,208 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,209 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:41,212 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,212 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:41,212 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:41,212 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,212 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:41,212 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,213 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,213 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831801250, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831801253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,255 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831801253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,312 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:41,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:41,314 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:41,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:41,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:41,315 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:41,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:41,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:41,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,316 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,317 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,317 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,318 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,318 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:41,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,318 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:41,318 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:41,318 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:41,319 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,319 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:41,319 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:41,319 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:41,319 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:41,322 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,322 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,322 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,322 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:41,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1971bf99[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:41,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:41,324 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,325 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:41,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:41,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:41,325 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,325 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:41,326 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:41,326 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,326 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,326 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:41,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:41,329 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:41,329 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:41,329 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:41,329 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:41,329 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:41,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,331 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:41,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:41,335 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:41,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,335 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:41,335 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,336 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:41,336 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,336 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,336 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,436 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:41,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:41,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:41,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:41,439 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:41,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:41,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:41,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,439 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:41,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,442 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:41,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:41,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:41,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:41,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:41,442 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:41,442 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,443 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:41,443 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:41,444 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,444 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:41,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,446 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5bb7c4ecbab14d1d811ac720b8e427b3 2023-03-26 11:55:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:41,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@483fb17[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:41,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:41,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,448 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,448 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:41,448 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:41,448 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:41,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,449 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:41,451 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:41,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,451 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:41,454 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:41,454 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:41,454 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:41,454 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:41,454 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:41,454 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:41,456 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,456 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,456 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:41,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:41,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=197 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/74b55e71f15246599ee10a1d10d191c2 2023-03-26 11:55:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,458 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:41,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:41,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,460 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:41,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:41,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,460 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,461 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:41,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:41,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/8f755ffa8091437aa2068586cfcabebb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8f755ffa8091437aa2068586cfcabebb 2023-03-26 11:55:41,461 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:41,461 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,461 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:41,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:41,461 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:41,462 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:41,462 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:41,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831801456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831801456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831801457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,465 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8f755ffa8091437aa2068586cfcabebb, entries=200, sequenceid=197, filesize=14.1 K 2023-03-26 11:55:41,466 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5bb7c4ecbab14d1d811ac720b8e427b3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5bb7c4ecbab14d1d811ac720b8e427b3 2023-03-26 11:55:41,469 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5bb7c4ecbab14d1d811ac720b8e427b3, entries=150, sequenceid=197, filesize=11.8 K 2023-03-26 11:55:41,470 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/74b55e71f15246599ee10a1d10d191c2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/74b55e71f15246599ee10a1d10d191c2 2023-03-26 11:55:41,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/74b55e71f15246599ee10a1d10d191c2, entries=150, sequenceid=197, filesize=11.8 K 2023-03-26 11:55:41,475 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e1f18589e0dd3700bbf1b6b539ec6e88 in 453ms, sequenceid=197, compaction requested=true 2023-03-26 11:55:41,475 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:41,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:41,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:41,475 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:41,475 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:41,475 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:41,476 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:41,476 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38670 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:41,476 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:41,476 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:41,476 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,476 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,476 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/8e32cfcdae1c4882b5ee9079c17953f9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/b11d00589bf0455b9210f6b5e4e22d0c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5bb7c4ecbab14d1d811ac720b8e427b3] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=35.4 K 2023-03-26 11:55:41,476 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/90617ece3b19443aaf3acf3f8a936aad, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ec9b3d9f44e648d48c09ed3fc620b9bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8f755ffa8091437aa2068586cfcabebb] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=37.8 K 2023-03-26 11:55:41,476 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8e32cfcdae1c4882b5ee9079c17953f9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679831738784 2023-03-26 11:55:41,476 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 90617ece3b19443aaf3acf3f8a936aad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679831738784 2023-03-26 11:55:41,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b11d00589bf0455b9210f6b5e4e22d0c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831738851 2023-03-26 11:55:41,477 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ec9b3d9f44e648d48c09ed3fc620b9bf, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831738851 2023-03-26 11:55:41,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5bb7c4ecbab14d1d811ac720b8e427b3, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831740010 2023-03-26 11:55:41,477 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8f755ffa8091437aa2068586cfcabebb, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831740010 2023-03-26 11:55:41,498 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#437 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:41,498 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#438 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:41,524 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/e714f834bc7a4a43a8d29db205e16719 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e714f834bc7a4a43a8d29db205e16719 2023-03-26 11:55:41,529 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into e714f834bc7a4a43a8d29db205e16719(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:41,529 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:41,529 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831741475; duration=0sec 2023-03-26 11:55:41,529 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:41,529 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:41,530 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36280 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:41,530 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:41,530 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,530 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5223938ea0fb4953830ab7f9c7a3685b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/217bda0b520c477ea95ba358b25eb838, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/74b55e71f15246599ee10a1d10d191c2] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=35.4 K 2023-03-26 11:55:41,531 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5223938ea0fb4953830ab7f9c7a3685b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=148, earliestPutTs=1679831738784 2023-03-26 11:55:41,531 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 217bda0b520c477ea95ba358b25eb838, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=183, earliestPutTs=1679831738851 2023-03-26 11:55:41,531 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 74b55e71f15246599ee10a1d10d191c2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831740010 2023-03-26 11:55:41,540 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#439 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:41,563 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:41,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:41,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:41,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:41,569 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:41,569 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:41,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:41,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:41,569 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/dbb2b3defdfb4a6cb6731bed7d338a12 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/dbb2b3defdfb4a6cb6731bed7d338a12 2023-03-26 11:55:41,569 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,570 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,571 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:41,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:41,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:41,571 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,572 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:41,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,572 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:41,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:41,573 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:41,575 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into dbb2b3defdfb4a6cb6731bed7d338a12(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:41,575 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:41,575 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831741475; duration=0sec 2023-03-26 11:55:41,575 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:41,576 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:41,576 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:41,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:41,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:41,576 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:41,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:41,580 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:41,580 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:41,581 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:41,622 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/b1b7c6df76c6471296699b188fb76bff 2023-03-26 11:55:41,769 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:41,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831801781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831801782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,795 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831801787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46648 deadline: 1679831801889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,894 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831801889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:41,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46634 deadline: 1679831801896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:41,913 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/500040e83eb649c0a6c2e431647298de as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/500040e83eb649c0a6c2e431647298de 2023-03-26 11:55:41,923 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into 500040e83eb649c0a6c2e431647298de(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:41,923 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:41,923 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831741475; duration=0sec 2023-03-26 11:55:41,923 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:42,038 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/84facc2ce281422081b4df00c578d440 2023-03-26 11:55:42,046 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=234 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/e11585c7790842898cf72d6ea3657653 2023-03-26 11:55:42,050 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/b1b7c6df76c6471296699b188fb76bff as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b1b7c6df76c6471296699b188fb76bff 2023-03-26 11:55:42,054 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b1b7c6df76c6471296699b188fb76bff, entries=150, sequenceid=234, filesize=11.8 K 2023-03-26 11:55:42,055 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/84facc2ce281422081b4df00c578d440 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/84facc2ce281422081b4df00c578d440 2023-03-26 11:55:42,061 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/84facc2ce281422081b4df00c578d440, entries=150, sequenceid=234, filesize=11.8 K 2023-03-26 11:55:42,061 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/e11585c7790842898cf72d6ea3657653 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/e11585c7790842898cf72d6ea3657653 2023-03-26 11:55:42,065 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/e11585c7790842898cf72d6ea3657653, entries=150, sequenceid=234, filesize=11.8 K 2023-03-26 11:55:42,066 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for e1f18589e0dd3700bbf1b6b539ec6e88 in 486ms, sequenceid=234, compaction requested=false 2023-03-26 11:55:42,066 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:42,066 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:42,066 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:42,066 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:42,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:42,067 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,075 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,075 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:42,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:42,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:42,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:42,077 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1788759a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:42,077 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:42,077 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,078 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,078 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:42,078 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:42,078 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:42,079 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,079 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:42,081 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:42,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:42,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:42,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:42,081 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:42,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:42,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:42,084 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:42,084 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:42,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:42,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:42,084 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:42,084 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:42,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,086 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:42,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:42,087 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:42,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,088 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:42,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,089 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:42,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:42,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:42,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:42,092 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:42,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:42,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:42,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:42,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:42,093 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:42,093 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:42,092 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:42,093 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:42,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:55:42,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:42,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:42,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:42,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:42,102 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:42,102 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:42,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/3222e17fe89c403c892bc29904afe4e4 2023-03-26 11:55:42,131 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/2c95c9b6d64e40e7be35f5ea3d3bc462 2023-03-26 11:55:42,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=248 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/6333394780f94f589aa0be9590d7c344 2023-03-26 11:55:42,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46700 deadline: 1679831802136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831802137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,147 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831802141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,148 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831802142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,153 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/3222e17fe89c403c892bc29904afe4e4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3222e17fe89c403c892bc29904afe4e4 2023-03-26 11:55:42,156 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3222e17fe89c403c892bc29904afe4e4, entries=250, sequenceid=248, filesize=16.4 K 2023-03-26 11:55:42,157 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/2c95c9b6d64e40e7be35f5ea3d3bc462 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2c95c9b6d64e40e7be35f5ea3d3bc462 2023-03-26 11:55:42,160 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2c95c9b6d64e40e7be35f5ea3d3bc462, entries=150, sequenceid=248, filesize=11.8 K 2023-03-26 11:55:42,161 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/6333394780f94f589aa0be9590d7c344 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6333394780f94f589aa0be9590d7c344 2023-03-26 11:55:42,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831802158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,164 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6333394780f94f589aa0be9590d7c344, entries=150, sequenceid=248, filesize=11.8 K 2023-03-26 11:55:42,165 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e1f18589e0dd3700bbf1b6b539ec6e88 in 64ms, sequenceid=248, compaction requested=true 2023-03-26 11:55:42,165 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:42,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:42,165 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:42,165 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:42,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:42,165 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:42,166 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 41162 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:42,166 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:42,166 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:42,166 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:42,166 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:42,166 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:42,166 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/500040e83eb649c0a6c2e431647298de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b1b7c6df76c6471296699b188fb76bff, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3222e17fe89c403c892bc29904afe4e4] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=40.2 K 2023-03-26 11:55:42,166 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e714f834bc7a4a43a8d29db205e16719, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/84facc2ce281422081b4df00c578d440, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2c95c9b6d64e40e7be35f5ea3d3bc462] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=35.5 K 2023-03-26 11:55:42,167 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 500040e83eb649c0a6c2e431647298de, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831740010 2023-03-26 11:55:42,167 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e714f834bc7a4a43a8d29db205e16719, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831740010 2023-03-26 11:55:42,167 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b1b7c6df76c6471296699b188fb76bff, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1679831741137 2023-03-26 11:55:42,167 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 84facc2ce281422081b4df00c578d440, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1679831741137 2023-03-26 11:55:42,168 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3222e17fe89c403c892bc29904afe4e4, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679831741780 2023-03-26 11:55:42,168 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2c95c9b6d64e40e7be35f5ea3d3bc462, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679831741780 2023-03-26 11:55:42,176 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#446 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:42,176 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#447 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:42,193 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:42,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:42,195 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:42,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:42,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:42,196 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:42,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:42,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:42,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:42,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:42,199 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:42,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:42,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:42,200 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:42,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:42,201 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:42,201 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:42,201 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:42,201 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:42,201 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:42,201 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:42,202 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:42,208 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/767a545909c74f059fe50894d75220d2 2023-03-26 11:55:42,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:42,251 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:42,272 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831802269, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831802270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,276 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831802272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831802373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831802377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831802377, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,587 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831802583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,588 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831802584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831802585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,592 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/6d1c2a10359b486d8eef324cebc4c970 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6d1c2a10359b486d8eef324cebc4c970 2023-03-26 11:55:42,593 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/fa0a86b0590145fc9be412da0fef4667 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/fa0a86b0590145fc9be412da0fef4667 2023-03-26 11:55:42,597 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 6d1c2a10359b486d8eef324cebc4c970(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:42,597 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into fa0a86b0590145fc9be412da0fef4667(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:42,597 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:42,597 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:42,597 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831742165; duration=0sec 2023-03-26 11:55:42,597 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831742165; duration=0sec 2023-03-26 11:55:42,597 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:42,597 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:42,597 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:42,598 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36382 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:42,598 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:42,598 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:42,598 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/dbb2b3defdfb4a6cb6731bed7d338a12, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/e11585c7790842898cf72d6ea3657653, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6333394780f94f589aa0be9590d7c344] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=35.5 K 2023-03-26 11:55:42,599 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dbb2b3defdfb4a6cb6731bed7d338a12, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=197, earliestPutTs=1679831740010 2023-03-26 11:55:42,599 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e11585c7790842898cf72d6ea3657653, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=234, earliestPutTs=1679831741137 2023-03-26 11:55:42,599 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6333394780f94f589aa0be9590d7c344, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679831741780 2023-03-26 11:55:42,604 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#450 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:42,611 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/9148885a96d74e51bcd69d51c687e323 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/9148885a96d74e51bcd69d51c687e323 2023-03-26 11:55:42,614 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into 9148885a96d74e51bcd69d51c687e323(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:42,614 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:42,614 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831742165; duration=0sec 2023-03-26 11:55:42,614 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:42,620 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/a6cd3bc31d604de3845ff3562e1848a2 2023-03-26 11:55:42,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831802889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831802891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:42,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:42,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831802894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,029 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=267 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/4a66b29660fd4607acf53af2f77e3698 2023-03-26 11:55:43,032 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/767a545909c74f059fe50894d75220d2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/767a545909c74f059fe50894d75220d2 2023-03-26 11:55:43,035 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/767a545909c74f059fe50894d75220d2, entries=150, sequenceid=267, filesize=11.9 K 2023-03-26 11:55:43,036 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/a6cd3bc31d604de3845ff3562e1848a2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a6cd3bc31d604de3845ff3562e1848a2 2023-03-26 11:55:43,039 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a6cd3bc31d604de3845ff3562e1848a2, entries=150, sequenceid=267, filesize=11.9 K 2023-03-26 11:55:43,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/4a66b29660fd4607acf53af2f77e3698 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4a66b29660fd4607acf53af2f77e3698 2023-03-26 11:55:43,043 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4a66b29660fd4607acf53af2f77e3698, entries=150, sequenceid=267, filesize=11.9 K 2023-03-26 11:55:43,043 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e1f18589e0dd3700bbf1b6b539ec6e88 in 842ms, sequenceid=267, compaction requested=false 2023-03-26 11:55:43,043 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:43,043 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:43,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:43,045 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,045 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:43,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:43,046 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,046 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:43,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:43,048 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38171578[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:43,048 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:43,048 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,049 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:43,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:43,049 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,049 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:43,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,051 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:43,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:43,051 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:43,051 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:43,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:43,053 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:43,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:43,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:43,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:43,054 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:43,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:43,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:43,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:43,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:43,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,058 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:43,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:43,059 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:43,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:43,059 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:43,059 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:43,059 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:43,059 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,159 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:43,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:43,161 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:43,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:43,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:43,162 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:43,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:43,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:43,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:43,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:43,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:43,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:43,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,165 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,166 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:43,166 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:43,166 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:43,166 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:43,166 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:43,166 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:43,166 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:55:43,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:43,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:43,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:43,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:43,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:43,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:43,175 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/eca05ea489094201b8f4d0d8956a2ac5 2023-03-26 11:55:43,185 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/e4218360a89d4516a6bbd9297f3a0f1d 2023-03-26 11:55:43,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:43,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:43,433 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831803426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,440 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831803431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,441 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831803431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,534 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831803534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831803542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,546 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831803542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,596 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=283 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/7701c04a52ea4cdb9c496eafb1b20211 2023-03-26 11:55:43,600 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/eca05ea489094201b8f4d0d8956a2ac5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/eca05ea489094201b8f4d0d8956a2ac5 2023-03-26 11:55:43,603 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/eca05ea489094201b8f4d0d8956a2ac5, entries=150, sequenceid=283, filesize=11.9 K 2023-03-26 11:55:43,603 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/e4218360a89d4516a6bbd9297f3a0f1d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e4218360a89d4516a6bbd9297f3a0f1d 2023-03-26 11:55:43,606 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e4218360a89d4516a6bbd9297f3a0f1d, entries=150, sequenceid=283, filesize=11.9 K 2023-03-26 11:55:43,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/7701c04a52ea4cdb9c496eafb1b20211 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7701c04a52ea4cdb9c496eafb1b20211 2023-03-26 11:55:43,610 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7701c04a52ea4cdb9c496eafb1b20211, entries=150, sequenceid=283, filesize=11.9 K 2023-03-26 11:55:43,611 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for e1f18589e0dd3700bbf1b6b539ec6e88 in 445ms, sequenceid=283, compaction requested=true 2023-03-26 11:55:43,611 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:43,611 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:43,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:43,613 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,613 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:43,614 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,614 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:43,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:43,615 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c8ef894[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:43,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:43,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,617 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:43,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:43,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:43,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:43,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:43,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,618 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:43,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:43,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:43,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:43,621 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:43,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:43,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:43,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:43,621 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:43,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:43,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,623 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:43,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,624 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:43,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:43,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:43,627 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:43,627 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:43,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:43,627 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:43,627 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:43,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:43,627 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:43,627 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:43,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:43,727 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:43,728 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:43,729 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:43,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:43,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:43,731 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:43,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:43,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:43,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:43,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,733 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:43,733 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:43,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:43,733 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,733 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:43,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:43,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:43,735 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:43,735 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:43,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:43,736 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:43,736 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-03-26 11:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:43,736 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:43,737 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:43,737 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:43,740 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:43,740 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:43,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831803758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,765 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831803761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,768 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831803762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831803866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,872 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831803866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:43,873 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:43,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831803869, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831804074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831804074, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,078 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831804075, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,143 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/c94af6b2b813406faecc6d006b88cce5 2023-03-26 11:55:44,161 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831804159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,163 DEBUG [Thread-2993] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4151 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:55:44,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831804164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,169 DEBUG [Thread-2989] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:55:44,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831804379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831804379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,387 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831804381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,552 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/01fb599a07724cb281914273c3e6819b 2023-03-26 11:55:44,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831804882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831804882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:44,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831804889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,966 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/aaf607974be1497b853e967e6c9209cc 2023-03-26 11:55:44,969 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/c94af6b2b813406faecc6d006b88cce5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/c94af6b2b813406faecc6d006b88cce5 2023-03-26 11:55:44,972 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/c94af6b2b813406faecc6d006b88cce5, entries=150, sequenceid=302, filesize=11.9 K 2023-03-26 11:55:44,973 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/01fb599a07724cb281914273c3e6819b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/01fb599a07724cb281914273c3e6819b 2023-03-26 11:55:44,976 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/01fb599a07724cb281914273c3e6819b, entries=150, sequenceid=302, filesize=11.9 K 2023-03-26 11:55:44,977 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/aaf607974be1497b853e967e6c9209cc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/aaf607974be1497b853e967e6c9209cc 2023-03-26 11:55:44,979 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/aaf607974be1497b853e967e6c9209cc, entries=150, sequenceid=302, filesize=11.9 K 2023-03-26 11:55:44,980 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1244ms, sequenceid=302, compaction requested=true 2023-03-26 11:55:44,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:44,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:44,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:44,982 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:44,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:44,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:44,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:44,984 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37cef551[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:44,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:44,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:44,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:44,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:44,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:44,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:44,985 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:44,995 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:44,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:44,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:44,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:44,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:44,997 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:44,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:44,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:44,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:44,998 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:44,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:44,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:44,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:44,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:44,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:45,003 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,003 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,003 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:45,004 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,004 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,104 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:45,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:45,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:45,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:45,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:45,108 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:45,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:45,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:45,109 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,109 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,113 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,113 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:45,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:45,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:45,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:45,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:45,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:45,116 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:45,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:45,126 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/8e42e18ccedc4adf87afab1b1cbe4d37 2023-03-26 11:55:45,141 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/98385d9eb72a4f6784741738d5394578 2023-03-26 11:55:45,554 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=315 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/5652a2b70fec451498ff2999a3616c6f 2023-03-26 11:55:45,558 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/8e42e18ccedc4adf87afab1b1cbe4d37 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8e42e18ccedc4adf87afab1b1cbe4d37 2023-03-26 11:55:45,561 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8e42e18ccedc4adf87afab1b1cbe4d37, entries=150, sequenceid=315, filesize=11.9 K 2023-03-26 11:55:45,562 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/98385d9eb72a4f6784741738d5394578 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/98385d9eb72a4f6784741738d5394578 2023-03-26 11:55:45,565 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/98385d9eb72a4f6784741738d5394578, entries=150, sequenceid=315, filesize=11.9 K 2023-03-26 11:55:45,566 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/5652a2b70fec451498ff2999a3616c6f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5652a2b70fec451498ff2999a3616c6f 2023-03-26 11:55:45,569 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5652a2b70fec451498ff2999a3616c6f, entries=150, sequenceid=315, filesize=11.9 K 2023-03-26 11:55:45,569 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for e1f18589e0dd3700bbf1b6b539ec6e88 in 453ms, sequenceid=315, compaction requested=true 2023-03-26 11:55:45,569 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:45,570 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:45,570 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:45,571 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,571 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,572 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:45,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@633e6297[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:45,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:45,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,574 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:45,574 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:45,575 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:45,575 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,575 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:45,577 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:45,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:45,579 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:45,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:45,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:45,579 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:45,579 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:45,581 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,581 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,581 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,581 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,582 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,585 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:45,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:45,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,585 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:45,586 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:45,686 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:45,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:45,688 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:45,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:45,689 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:45,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:45,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:45,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,689 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,690 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,691 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:45,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:45,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:45,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:45,692 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:45,692 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:45,694 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,694 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,694 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,695 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:45,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6382075d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:45,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:45,697 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:45,698 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:45,698 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,698 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:45,699 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,699 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:45,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:45,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:45,702 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:45,703 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:45,703 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:45,703 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:45,703 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:45,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,704 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,705 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,706 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:45,709 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:45,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:45,708 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,709 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:45,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,709 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:45,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,709 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,809 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:45,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:45,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:45,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:45,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:45,829 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:45,829 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,830 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,833 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,833 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,833 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,833 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:45,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:45,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,835 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:45,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:45,836 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:45,836 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,836 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:45,836 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:45,836 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:45,836 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:45,838 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,838 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,839 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,839 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:45,840 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@381319e1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:45,840 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:45,840 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:45,842 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:45,844 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:45,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,845 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:45,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:45,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:45,851 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:45,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:45,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:45,851 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:45,851 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:45,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,858 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,862 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,862 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:45,862 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:45,864 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:45,864 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:45,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:45,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:55:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:45,908 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:45,914 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/b9ca0404ae5048eebb6bb6b6e1669812 2023-03-26 11:55:45,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831745944, max timestamp=1679831745961Num uniques 859; 2023-03-26 11:55:45,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:55:45,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831745944, max timestamp=1679831745961Num uniques 859; 2023-03-26 11:55:45,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:55:45,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1679831745944, max timestamp=1679831745961Num uniques 859; 2023-03-26 11:55:45,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-03-26 11:55:45,964 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:45,966 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:45,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:45,967 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:45,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:45,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:45,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,969 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,969 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:45,970 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:45,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:45,971 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:45,971 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:45,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:45,971 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:45,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:45,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:45,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:45,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:45,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:45,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78b717cf[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:45,977 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:45,977 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:45,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:45,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:45,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:45,980 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:45,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:45,982 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:45,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:45,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:45,982 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:45,982 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:45,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,983 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:45,983 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:45,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:45,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:45,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:45,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:45,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:45,989 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:45,989 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:45,989 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:45,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:45,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:45,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:45,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:45,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831806003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831806006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,020 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831806015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,090 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:46,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:46,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:46,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:46,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:46,093 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:46,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:46,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:46,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,093 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,094 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:46,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:46,095 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:46,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:46,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:46,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:46,096 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:46,096 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:46,098 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:46,099 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680de894[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:46,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:46,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,100 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,100 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:46,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:46,101 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:46,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:46,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:46,103 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,103 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:46,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:46,105 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:46,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:46,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:46,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:46,105 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:46,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,108 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,109 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:46,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,111 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,111 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:46,111 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,111 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:46,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,112 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:46,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831806116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831806120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,127 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831806120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,211 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:46,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:46,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:46,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:46,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:46,214 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:46,214 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:46,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:46,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,215 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,216 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,216 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,216 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,216 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:46,217 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:46,217 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:46,217 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:46,218 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,218 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:46,218 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:46,218 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:46,218 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:46,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,220 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,221 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:46,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bc6a227[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:46,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:46,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,223 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:46,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:46,223 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:46,225 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,225 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:46,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:46,227 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:46,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:46,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:46,227 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:46,227 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:46,228 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,228 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,229 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,229 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,230 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,236 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,236 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:46,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:46,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:46,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,236 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:46,237 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:46,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/384ce3f052f4403c8ab0726c55d78bcf 2023-03-26 11:55:46,330 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831806323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,337 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:46,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:46,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:46,343 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831806329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831806334, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:46,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:46,372 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:46,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:46,372 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:46,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,375 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,375 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:46,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:46,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,378 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:46,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:46,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,384 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:46,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:46,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:46,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:46,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:46,398 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=328 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/057259c67b624bf1bd3541222f159541 2023-03-26 11:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:46,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b3a7caf[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:46,401 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:46,401 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,403 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/b9ca0404ae5048eebb6bb6b6e1669812 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b9ca0404ae5048eebb6bb6b6e1669812 2023-03-26 11:55:46,403 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:46,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:46,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:46,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,404 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:46,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:46,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,405 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:46,405 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,408 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b9ca0404ae5048eebb6bb6b6e1669812, entries=300, sequenceid=328, filesize=19.0 K 2023-03-26 11:55:46,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,409 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/384ce3f052f4403c8ab0726c55d78bcf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/384ce3f052f4403c8ab0726c55d78bcf 2023-03-26 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:46,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:46,410 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:46,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:46,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:46,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:46,411 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:46,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,412 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/384ce3f052f4403c8ab0726c55d78bcf, entries=150, sequenceid=328, filesize=11.9 K 2023-03-26 11:55:46,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,412 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,413 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,415 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,415 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/057259c67b624bf1bd3541222f159541 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/057259c67b624bf1bd3541222f159541 2023-03-26 11:55:46,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,418 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,418 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,418 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:46,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:46,418 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,418 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:46,419 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:46,418 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,419 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:46,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/057259c67b624bf1bd3541222f159541, entries=150, sequenceid=328, filesize=11.9 K 2023-03-26 11:55:46,422 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for e1f18589e0dd3700bbf1b6b539ec6e88 in 516ms, sequenceid=328, compaction requested=true 2023-03-26 11:55:46,423 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:46,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:46,423 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:46,423 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:46,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:46,423 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:46,424 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 80662 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:46,424 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73342 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:46,424 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:46,424 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:46,424 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,424 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,424 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/fa0a86b0590145fc9be412da0fef4667, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/767a545909c74f059fe50894d75220d2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/eca05ea489094201b8f4d0d8956a2ac5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/c94af6b2b813406faecc6d006b88cce5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8e42e18ccedc4adf87afab1b1cbe4d37, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b9ca0404ae5048eebb6bb6b6e1669812] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=78.8 K 2023-03-26 11:55:46,424 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6d1c2a10359b486d8eef324cebc4c970, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a6cd3bc31d604de3845ff3562e1848a2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e4218360a89d4516a6bbd9297f3a0f1d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/01fb599a07724cb281914273c3e6819b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/98385d9eb72a4f6784741738d5394578, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/384ce3f052f4403c8ab0726c55d78bcf] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=71.6 K 2023-03-26 11:55:46,425 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting fa0a86b0590145fc9be412da0fef4667, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679831741780 2023-03-26 11:55:46,425 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6d1c2a10359b486d8eef324cebc4c970, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679831741780 2023-03-26 11:55:46,425 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 767a545909c74f059fe50894d75220d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679831742130 2023-03-26 11:55:46,425 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a6cd3bc31d604de3845ff3562e1848a2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679831742130 2023-03-26 11:55:46,426 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting eca05ea489094201b8f4d0d8956a2ac5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679831742256 2023-03-26 11:55:46,426 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e4218360a89d4516a6bbd9297f3a0f1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679831742256 2023-03-26 11:55:46,426 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c94af6b2b813406faecc6d006b88cce5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679831743419 2023-03-26 11:55:46,426 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 01fb599a07724cb281914273c3e6819b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679831743419 2023-03-26 11:55:46,426 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8e42e18ccedc4adf87afab1b1cbe4d37, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679831743750 2023-03-26 11:55:46,427 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 98385d9eb72a4f6784741738d5394578, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679831743750 2023-03-26 11:55:46,427 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b9ca0404ae5048eebb6bb6b6e1669812, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679831745897 2023-03-26 11:55:46,427 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 384ce3f052f4403c8ab0726c55d78bcf, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679831745903 2023-03-26 11:55:46,435 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#464 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:46,438 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#465 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:46,519 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:46,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:46,521 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:46,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:46,522 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:46,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:46,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:46,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,525 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:46,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,525 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,526 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:46,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:46,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:46,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:46,527 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:46,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:46,527 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-03-26 11:55:46,527 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:46,528 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:46,535 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/a54a33d1c38846d3925e5f2ebad52ee6 2023-03-26 11:55:46,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:46,639 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:46,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831806661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,670 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831806667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831806668, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831806771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831806771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:46,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831806781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,845 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/a99febd4cac94f349ef7920e6040e681 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a99febd4cac94f349ef7920e6040e681 2023-03-26 11:55:46,849 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into a99febd4cac94f349ef7920e6040e681(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:46,849 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:46,849 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=10, startTime=1679831746423; duration=0sec 2023-03-26 11:55:46,849 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:46,849 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:55:46,850 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 73342 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:55:46,851 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:46,851 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,851 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/9148885a96d74e51bcd69d51c687e323, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4a66b29660fd4607acf53af2f77e3698, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7701c04a52ea4cdb9c496eafb1b20211, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/aaf607974be1497b853e967e6c9209cc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5652a2b70fec451498ff2999a3616c6f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/057259c67b624bf1bd3541222f159541] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=71.6 K 2023-03-26 11:55:46,851 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9148885a96d74e51bcd69d51c687e323, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=248, earliestPutTs=1679831741780 2023-03-26 11:55:46,851 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 4a66b29660fd4607acf53af2f77e3698, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=267, earliestPutTs=1679831742130 2023-03-26 11:55:46,852 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 7701c04a52ea4cdb9c496eafb1b20211, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=283, earliestPutTs=1679831742256 2023-03-26 11:55:46,852 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting aaf607974be1497b853e967e6c9209cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1679831743419 2023-03-26 11:55:46,852 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 5652a2b70fec451498ff2999a3616c6f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=315, earliestPutTs=1679831743750 2023-03-26 11:55:46,853 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 057259c67b624bf1bd3541222f159541, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679831745903 2023-03-26 11:55:46,853 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/2f148b4f381749848573f8113203b46e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2f148b4f381749848573f8113203b46e 2023-03-26 11:55:46,857 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 2f148b4f381749848573f8113203b46e(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:46,857 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:46,857 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=10, startTime=1679831746423; duration=0sec 2023-03-26 11:55:46,857 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:46,862 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#468 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:46,869 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/31556b31214c4e349858ea9714d1b1a3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/31556b31214c4e349858ea9714d1b1a3 2023-03-26 11:55:46,873 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into 31556b31214c4e349858ea9714d1b1a3(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:46,873 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:46,873 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=10, startTime=1679831746423; duration=0sec 2023-03-26 11:55:46,873 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:46,948 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/dcbfb8498a174c02939a0f61d1572bac 2023-03-26 11:55:46,958 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/67b63939728a45228aae94d6ae2bfaa8 2023-03-26 11:55:46,962 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/a54a33d1c38846d3925e5f2ebad52ee6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a54a33d1c38846d3925e5f2ebad52ee6 2023-03-26 11:55:46,965 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a54a33d1c38846d3925e5f2ebad52ee6, entries=150, sequenceid=362, filesize=11.9 K 2023-03-26 11:55:46,966 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/dcbfb8498a174c02939a0f61d1572bac as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dcbfb8498a174c02939a0f61d1572bac 2023-03-26 11:55:46,969 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dcbfb8498a174c02939a0f61d1572bac, entries=150, sequenceid=362, filesize=11.9 K 2023-03-26 11:55:46,969 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/67b63939728a45228aae94d6ae2bfaa8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/67b63939728a45228aae94d6ae2bfaa8 2023-03-26 11:55:46,972 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/67b63939728a45228aae94d6ae2bfaa8, entries=150, sequenceid=362, filesize=11.9 K 2023-03-26 11:55:46,973 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for e1f18589e0dd3700bbf1b6b539ec6e88 in 446ms, sequenceid=362, compaction requested=false 2023-03-26 11:55:46,973 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:46,973 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:46,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:46,974 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:46,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,976 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:46,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7bf30d68[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:46,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:46,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,979 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,979 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:46,980 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:55:46,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,980 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:46,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:46,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:46,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:46,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:46,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:46,980 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:46,980 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:46,981 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,981 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:46,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,982 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:46,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:46,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:46,986 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:46,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:46,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:46,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:46,986 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:46,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:46,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:46,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:46,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,991 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:46,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:46,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:46,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:46,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:46,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:46,993 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:46,993 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:46,993 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:46,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:46,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831807040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831807041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,050 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831807041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,094 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,096 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,097 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,099 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,099 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,100 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:47,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,102 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,102 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,102 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,102 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,102 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,104 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c3d03ca[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,105 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,106 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,110 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,120 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,126 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,126 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,126 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,126 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,128 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,135 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,141 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,142 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,142 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,142 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831807149, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831807151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831807151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,242 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,245 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,245 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,246 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,247 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,248 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,251 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,252 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@524451f0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,252 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,253 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,254 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,254 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,254 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,259 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,259 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,259 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,259 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,261 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,261 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,264 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,265 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,264 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,265 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,265 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,265 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,265 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,265 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,265 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,265 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,266 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:47,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831807353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831807360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,365 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,366 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831807362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,368 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,368 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,370 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,370 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,370 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:47,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,371 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,372 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,372 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,374 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,374 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3af38013[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,377 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,377 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,379 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,381 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,382 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,383 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,383 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,383 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,384 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,385 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,388 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/61017bbc872342f0a17c6d719f139312 2023-03-26 11:55:47,388 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,388 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,388 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,388 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:47,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,389 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,489 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,492 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,492 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,495 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:47,495 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,495 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,496 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,496 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,496 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,496 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,496 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,497 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59885268[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,500 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,501 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,501 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,503 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,505 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,505 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,511 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,511 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,511 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,511 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,511 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:47,611 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,617 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,619 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,620 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,621 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:47,621 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,636 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,636 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,636 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,636 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,636 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,642 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,642 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,653 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dbf39c0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,653 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,657 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,657 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,657 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,659 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,662 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831807659, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,664 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,664 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,664 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,665 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,669 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831807665, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:47,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831807669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,675 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,677 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,678 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:47,678 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,678 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,678 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,778 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,781 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,783 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,784 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,784 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:47,784 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d043a56[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,791 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,791 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,793 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,795 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,795 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,795 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,795 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,796 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,797 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/afb75c0fa9b54e7f868d9a4dea280832 2023-03-26 11:55:47,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,798 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,799 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,802 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,801 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,802 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,802 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,802 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:47,902 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:47,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:47,904 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:47,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:47,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:47,905 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:47,905 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,905 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:47,905 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:47,906 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,906 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,908 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:47,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:47,909 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:47,909 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:47,910 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:47,910 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:47,912 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,912 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:47,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:47,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b51c6fd[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:47,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:47,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:47,915 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,915 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:47,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:47,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:47,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:47,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:47,919 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:47,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:47,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:47,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:47,919 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:47,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:47,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,922 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,923 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:47,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:47,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:47,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:47,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:47,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:47,926 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:47,925 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:47,926 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:47,926 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:47,926 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:47,926 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:47,926 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:47,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:47,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:47,927 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:48,026 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:48,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:48,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:48,030 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:48,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:48,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:48,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:48,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:48,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:48,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:48,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:48,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:48,034 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:48,034 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:48,034 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:48,034 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:48,036 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,036 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,037 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:48,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:48,039 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d1cced[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:48,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:48,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:48,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:48,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:48,044 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:48,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:48,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:48,046 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:48,046 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:48,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:48,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:48,046 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:48,046 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:48,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:48,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:48,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,049 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:48,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,050 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,052 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,052 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:48,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:48,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:48,052 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:48,053 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:48,053 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:48,053 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:48,053 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:48,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:48,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:48,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:48,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:48,153 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:48,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:48,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:48,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:48,156 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:48,156 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:48,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:48,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:48,157 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:48,161 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,161 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,163 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:48,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:48,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:48,164 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:48,165 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:48,165 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:48,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:48,165 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:48,167 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,167 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:48,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,168 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:48,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3494940e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:48,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:48,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831808163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,170 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:48,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:48,170 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,171 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:48,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:48,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:48,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:48,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:48,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:48,175 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:48,175 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:48,175 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:48,176 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:48,176 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:48,177 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:48,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831808174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:48,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831808175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,181 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,181 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:48,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,181 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:48,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:48,182 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:48,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,186 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,186 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:48,186 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:48,186 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:48,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:48,186 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:48,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:48,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:48,187 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:48,187 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:48,187 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:48,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:48,190 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:48,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46702 deadline: 1679831808187, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,192 DEBUG [Thread-2993] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8180 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:55:48,206 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=377 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/0feb93b31ffd42b2a03be5dc418de4f7 2023-03-26 11:55:48,210 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/61017bbc872342f0a17c6d719f139312 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/61017bbc872342f0a17c6d719f139312 2023-03-26 11:55:48,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:48,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46682 deadline: 1679831808207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,212 DEBUG [Thread-2989] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8194 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:55:48,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/61017bbc872342f0a17c6d719f139312, entries=200, sequenceid=377, filesize=14.3 K 2023-03-26 11:55:48,216 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/afb75c0fa9b54e7f868d9a4dea280832 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/afb75c0fa9b54e7f868d9a4dea280832 2023-03-26 11:55:48,219 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/afb75c0fa9b54e7f868d9a4dea280832, entries=150, sequenceid=377, filesize=11.9 K 2023-03-26 11:55:48,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/0feb93b31ffd42b2a03be5dc418de4f7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0feb93b31ffd42b2a03be5dc418de4f7 2023-03-26 11:55:48,223 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0feb93b31ffd42b2a03be5dc418de4f7, entries=150, sequenceid=377, filesize=11.9 K 2023-03-26 11:55:48,224 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1244ms, sequenceid=377, compaction requested=true 2023-03-26 11:55:48,224 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:48,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:48,224 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:48,224 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:48,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:48,224 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:48,225 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39578 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:48,225 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:48,225 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,225 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a99febd4cac94f349ef7920e6040e681, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a54a33d1c38846d3925e5f2ebad52ee6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/61017bbc872342f0a17c6d719f139312] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=38.7 K 2023-03-26 11:55:48,226 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:48,226 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:48,226 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,226 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2f148b4f381749848573f8113203b46e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dcbfb8498a174c02939a0f61d1572bac, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/afb75c0fa9b54e7f868d9a4dea280832] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.3 K 2023-03-26 11:55:48,226 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a99febd4cac94f349ef7920e6040e681, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679831745903 2023-03-26 11:55:48,226 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2f148b4f381749848573f8113203b46e, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679831745903 2023-03-26 11:55:48,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a54a33d1c38846d3925e5f2ebad52ee6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679831745995 2023-03-26 11:55:48,227 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dcbfb8498a174c02939a0f61d1572bac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679831745995 2023-03-26 11:55:48,227 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting afb75c0fa9b54e7f868d9a4dea280832, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679831746659 2023-03-26 11:55:48,227 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 61017bbc872342f0a17c6d719f139312, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679831746652 2023-03-26 11:55:48,233 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:48,235 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:48,248 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/07944f23c79a477fa99898663a7bc7b0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/07944f23c79a477fa99898663a7bc7b0 2023-03-26 11:55:48,253 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into 07944f23c79a477fa99898663a7bc7b0(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:48,253 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:48,253 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831748224; duration=0sec 2023-03-26 11:55:48,253 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:48,253 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:48,254 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:48,254 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:48,254 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,254 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/31556b31214c4e349858ea9714d1b1a3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/67b63939728a45228aae94d6ae2bfaa8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0feb93b31ffd42b2a03be5dc418de4f7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.3 K 2023-03-26 11:55:48,255 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 31556b31214c4e349858ea9714d1b1a3, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=328, earliestPutTs=1679831745903 2023-03-26 11:55:48,255 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 67b63939728a45228aae94d6ae2bfaa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1679831745995 2023-03-26 11:55:48,255 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0feb93b31ffd42b2a03be5dc418de4f7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679831746659 2023-03-26 11:55:48,263 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#475 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:48,276 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/de79ef26006b4975bf47bd05093c3122 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/de79ef26006b4975bf47bd05093c3122 2023-03-26 11:55:48,282 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into de79ef26006b4975bf47bd05093c3122(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:48,282 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:48,282 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831748224; duration=0sec 2023-03-26 11:55:48,282 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:48,287 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:48,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:48,289 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:48,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:48,290 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:48,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:48,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:48,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:48,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:48,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:48,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:48,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:48,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:48,293 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:48,293 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:48,293 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:48,293 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:48,294 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:48,294 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:48,294 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:48,648 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/36a5b663cac24554848b1eb91002a800 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/36a5b663cac24554848b1eb91002a800 2023-03-26 11:55:48,652 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 36a5b663cac24554848b1eb91002a800(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:48,652 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:48,652 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831748224; duration=0sec 2023-03-26 11:55:48,652 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:48,702 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/33967de1c74343e992a5a1a833085bbd 2023-03-26 11:55:49,114 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5d01f750666643f097590f0e0275cbe9 2023-03-26 11:55:49,123 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=399 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/fa822dc540c94a90a48245ff126a6b51 2023-03-26 11:55:49,126 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/33967de1c74343e992a5a1a833085bbd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/33967de1c74343e992a5a1a833085bbd 2023-03-26 11:55:49,130 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/33967de1c74343e992a5a1a833085bbd, entries=150, sequenceid=399, filesize=11.9 K 2023-03-26 11:55:49,131 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5d01f750666643f097590f0e0275cbe9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5d01f750666643f097590f0e0275cbe9 2023-03-26 11:55:49,138 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5d01f750666643f097590f0e0275cbe9, entries=150, sequenceid=399, filesize=11.9 K 2023-03-26 11:55:49,139 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/fa822dc540c94a90a48245ff126a6b51 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/fa822dc540c94a90a48245ff126a6b51 2023-03-26 11:55:49,141 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/fa822dc540c94a90a48245ff126a6b51, entries=150, sequenceid=399, filesize=11.9 K 2023-03-26 11:55:49,142 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for e1f18589e0dd3700bbf1b6b539ec6e88 in 848ms, sequenceid=399, compaction requested=false 2023-03-26 11:55:49,142 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:49,142 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,142 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,144 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,146 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28dede40[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,146 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,146 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,147 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,148 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,149 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,151 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,151 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,151 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,154 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,155 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,158 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,158 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,159 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,159 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:49,186 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:55:49,186 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:49,187 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:49,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831749198, max timestamp=1679831749219Num uniques 859; 2023-03-26 11:55:49,220 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:55:49,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831749198, max timestamp=1679831749219Num uniques 859; 2023-03-26 11:55:49,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831749198, max timestamp=1679831749219Num uniques 859; 2023-03-26 11:55:49,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:55:49,221 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:55:49,259 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:49,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:49,261 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:49,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:49,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:49,262 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:49,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:49,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:49,263 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,263 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,264 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,264 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,264 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:49,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,265 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,265 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:49,265 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:49,265 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,265 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:49,266 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,266 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:49,266 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:49,266 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,266 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,268 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831809261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,268 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d2d9388[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831809262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831809263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,271 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,272 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,272 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,272 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,273 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,273 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,275 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,275 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,275 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,280 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,280 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,280 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,281 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,285 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,285 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831809369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831809371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,374 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831809371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,385 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:49,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:49,387 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:49,388 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:49,388 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:49,388 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:49,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:49,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,389 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,391 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:49,392 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:49,392 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:49,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:49,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:49,392 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,392 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,394 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33959664[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,397 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,397 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,397 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,400 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,400 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,402 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,402 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,406 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,408 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:49,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,408 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,408 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,408 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,508 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:49,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:49,511 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:49,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:49,512 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:49,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:49,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:49,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,513 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,514 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,514 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,514 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,515 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:49,515 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:49,515 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:49,515 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,515 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:49,515 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:49,515 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,515 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,516 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,519 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,519 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23b150fc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,520 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,522 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,523 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,524 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,526 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,526 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,526 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,526 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,528 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,528 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,528 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,530 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,531 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,533 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,533 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,533 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,533 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,534 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,534 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,534 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831809576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831809576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831809576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,593 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/0b5de64a20bd44b48061a33b868e9f8c 2023-03-26 11:55:49,634 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:49,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:49,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:49,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:49,637 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:49,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:49,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:49,638 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,638 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,639 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,639 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:49,640 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,640 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:49,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:49,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:49,640 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,641 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:49,642 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,642 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:49,642 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:49,642 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,642 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,643 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,644 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,646 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,647 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d43b614[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,649 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,650 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,650 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,650 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,652 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,654 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,654 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,654 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,656 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,657 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,662 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,663 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,662 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,663 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,663 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:49,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,763 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:49,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:49,765 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:49,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:49,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:49,767 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:49,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:49,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:49,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,773 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,774 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:49,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:49,775 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:49,775 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:49,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:49,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:49,775 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,775 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,777 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c23dbe7[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,778 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,779 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,780 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,780 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,781 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,784 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,784 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,784 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,787 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,788 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,790 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,790 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:49,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,791 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,791 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831809880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,884 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831809880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:49,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831809881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,891 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:49,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:49,893 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:49,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:49,894 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:49,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:49,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:49,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,895 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,895 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,896 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,896 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,896 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:49,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,896 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:49,897 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:49,897 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:49,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:49,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:49,897 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:49,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:49,899 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:49,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@331a5eb1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:49,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:49,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,904 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:49,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:49,904 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:49,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,906 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:49,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:49,908 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:49,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:49,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:49,908 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:49,908 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:49,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:49,910 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:49,911 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:49,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:49,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:49,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:49,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:49,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:49,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:49,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:49,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:49,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:49,914 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:49,914 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:49,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:49,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:49,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:50,006 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/6dce5d2bc4af4509bbaf21bf51877522 2023-03-26 11:55:50,014 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:50,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:50,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:50,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:50,018 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:50,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:50,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:50,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:50,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,018 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=410 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/3a908b3973f4497e912a1ad6ce745ca8 2023-03-26 11:55:50,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:50,020 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:50,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:50,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:50,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:50,022 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/0b5de64a20bd44b48061a33b868e9f8c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/0b5de64a20bd44b48061a33b868e9f8c 2023-03-26 11:55:50,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:50,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:50,023 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,023 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:50,023 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:50,023 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:50,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:50,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:50,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:50,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/0b5de64a20bd44b48061a33b868e9f8c, entries=200, sequenceid=410, filesize=14.3 K 2023-03-26 11:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:50,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:50,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1a05ce9e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:50,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:50,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/6dce5d2bc4af4509bbaf21bf51877522 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6dce5d2bc4af4509bbaf21bf51877522 2023-03-26 11:55:50,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:50,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:50,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:50,028 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,028 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:50,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:50,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:50,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:50,030 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6dce5d2bc4af4509bbaf21bf51877522, entries=150, sequenceid=410, filesize=11.9 K 2023-03-26 11:55:50,031 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/3a908b3973f4497e912a1ad6ce745ca8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3a908b3973f4497e912a1ad6ce745ca8 2023-03-26 11:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:50,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:50,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:50,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:50,034 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:50,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:50,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:50,034 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:50,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3a908b3973f4497e912a1ad6ce745ca8, entries=150, sequenceid=410, filesize=11.9 K 2023-03-26 11:55:50,034 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:50,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for e1f18589e0dd3700bbf1b6b539ec6e88 in 849ms, sequenceid=410, compaction requested=true 2023-03-26 11:55:50,035 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:50,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:50,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:50,035 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:50,035 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:50,035 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:50,036 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:50,037 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39680 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:50,037 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:50,037 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:50,037 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,037 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,037 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/36a5b663cac24554848b1eb91002a800, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5d01f750666643f097590f0e0275cbe9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6dce5d2bc4af4509bbaf21bf51877522] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.4 K 2023-03-26 11:55:50,037 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/07944f23c79a477fa99898663a7bc7b0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/33967de1c74343e992a5a1a833085bbd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/0b5de64a20bd44b48061a33b868e9f8c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=38.8 K 2023-03-26 11:55:50,037 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 36a5b663cac24554848b1eb91002a800, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679831746659 2023-03-26 11:55:50,037 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 07944f23c79a477fa99898663a7bc7b0, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679831746659 2023-03-26 11:55:50,038 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5d01f750666643f097590f0e0275cbe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679831747038 2023-03-26 11:55:50,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,038 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 33967de1c74343e992a5a1a833085bbd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679831747038 2023-03-26 11:55:50,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,038 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:50,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:50,038 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6dce5d2bc4af4509bbaf21bf51877522, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679831749179 2023-03-26 11:55:50,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,039 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0b5de64a20bd44b48061a33b868e9f8c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679831749179 2023-03-26 11:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:50,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:50,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:50,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:50,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:50,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:50,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:50,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:50,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:50,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:50,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:50,047 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:50,048 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,048 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:50,048 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:50,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:50,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:50,055 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:50,055 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#483 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:50,148 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:50,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:50,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:55:50,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:55:50,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:55:50,151 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:55:50,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:50,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:55:50,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:55:50,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,151 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,153 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:50,153 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:50,153 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,153 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:55:50,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,154 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:50,154 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:55:50,154 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:55:50,154 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:55:50,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:55:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:55:50,158 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:55:50,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. started... 2023-03-26 11:55:50,158 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=134.18 KB heapSize=380.30 KB 2023-03-26 11:55:50,158 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:50,159 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:50,159 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:50,159 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:50,159 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:50,159 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:55:50,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:50,393 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. as already flushing 2023-03-26 11:55:50,405 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831810403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831810403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831810404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,473 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/151ab22567934ed3908167c3b864f681 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/151ab22567934ed3908167c3b864f681 2023-03-26 11:55:50,473 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/607483c1203948089b40500aed22a904 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/607483c1203948089b40500aed22a904 2023-03-26 11:55:50,477 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 151ab22567934ed3908167c3b864f681(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:50,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:50,477 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831750035; duration=0sec 2023-03-26 11:55:50,477 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into 607483c1203948089b40500aed22a904(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:50,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:50,477 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:50,477 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:50,477 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831750035; duration=0sec 2023-03-26 11:55:50,477 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:50,478 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:50,478 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:50,478 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,478 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/de79ef26006b4975bf47bd05093c3122, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/fa822dc540c94a90a48245ff126a6b51, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3a908b3973f4497e912a1ad6ce745ca8] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.4 K 2023-03-26 11:55:50,478 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting de79ef26006b4975bf47bd05093c3122, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=377, earliestPutTs=1679831746659 2023-03-26 11:55:50,479 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fa822dc540c94a90a48245ff126a6b51, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=399, earliestPutTs=1679831747038 2023-03-26 11:55:50,479 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3a908b3973f4497e912a1ad6ce745ca8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679831749179 2023-03-26 11:55:50,483 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#485 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:50,510 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831810506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831810512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831810512, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,566 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/ddd1559c958f450f9cd7f6a08b91ab15 2023-03-26 11:55:50,575 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/7e17c474d7604f29aaeac50b499471d2 2023-03-26 11:55:50,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46700 deadline: 1679831810712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46634 deadline: 1679831810718, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:55:50,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46648 deadline: 1679831810719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,754 DEBUG [Thread-3008] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x10a344e6 to 127.0.0.1:53955 2023-03-26 11:55:50,754 DEBUG [Thread-3008] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:50,756 DEBUG [Thread-3006] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed183a4 to 127.0.0.1:53955 2023-03-26 11:55:50,756 DEBUG [Thread-3006] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:50,757 DEBUG [Thread-3004] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x549bef45 to 127.0.0.1:53955 2023-03-26 11:55:50,757 DEBUG [Thread-3004] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:50,760 DEBUG [Thread-3000] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63093e21 to 127.0.0.1:53955 2023-03-26 11:55:50,760 DEBUG [Thread-3000] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:50,761 DEBUG [Thread-3002] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0b475d8d to 127.0.0.1:53955 2023-03-26 11:55:50,761 DEBUG [Thread-3002] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:50,890 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/df553d06cc414045aa1adf78c8f195bd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/df553d06cc414045aa1adf78c8f195bd 2023-03-26 11:55:50,894 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into df553d06cc414045aa1adf78c8f195bd(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:50,894 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:50,894 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831750035; duration=0sec 2023-03-26 11:55:50,894 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:50,985 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/6393e0c7babc491c97811547b07996c9 2023-03-26 11:55:50,989 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/ddd1559c958f450f9cd7f6a08b91ab15 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ddd1559c958f450f9cd7f6a08b91ab15 2023-03-26 11:55:50,991 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ddd1559c958f450f9cd7f6a08b91ab15, entries=150, sequenceid=446, filesize=11.9 K 2023-03-26 11:55:50,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/7e17c474d7604f29aaeac50b499471d2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/7e17c474d7604f29aaeac50b499471d2 2023-03-26 11:55:50,995 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/7e17c474d7604f29aaeac50b499471d2, entries=150, sequenceid=446, filesize=11.9 K 2023-03-26 11:55:50,995 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/6393e0c7babc491c97811547b07996c9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6393e0c7babc491c97811547b07996c9 2023-03-26 11:55:50,998 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6393e0c7babc491c97811547b07996c9, entries=150, sequenceid=446, filesize=11.9 K 2023-03-26 11:55:50,998 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~380.25 KB/389376, currentSize=46.96 KB/48090 for e1f18589e0dd3700bbf1b6b539ec6e88 in 840ms, sequenceid=446, compaction requested=false 2023-03-26 11:55:50,998 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:50,998 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:55:50,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:55:50,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:50,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:55:50,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,002 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:51,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:55:51,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a9638dc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:55:51,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:55:51,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,004 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:55:51,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:55:51,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:55:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:55:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:51,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:51,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:51,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:55:51,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:55:51,008 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:55:51,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:55:51,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:55:51,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:55:51,008 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:55:51,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:55:51,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:51,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:51,010 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:55:51,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:51,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:55:51,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,014 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,014 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:55:51,014 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:55:51,014 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:51,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:55:51,014 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:55:51,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:51,014 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:55:51,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:55:51,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:55:51,014 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:55:51,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:55:51,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:55:51,015 DEBUG [Thread-2999] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:55:51,015 DEBUG [Thread-2999] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:55:51,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:55:51,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:55:51,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:51,018 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:55:51,018 DEBUG [Thread-2991] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x50002343 to 127.0.0.1:53955 2023-03-26 11:55:51,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:51,018 DEBUG [Thread-2991] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:51,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:51,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:51,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:51,018 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:51,018 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:51,023 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/6b21eb239566489abcd6fd0ba1163e61 2023-03-26 11:55:51,024 DEBUG [Thread-2997] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x70e2be04 to 127.0.0.1:53955 2023-03-26 11:55:51,024 DEBUG [Thread-2997] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:51,026 DEBUG [Thread-2995] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6f2b3c03 to 127.0.0.1:53955 2023-03-26 11:55:51,026 DEBUG [Thread-2995] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:51,115 DEBUG [Thread-2999] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:55:51,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:55:51,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5fe385a08d874654967eb4993367e648 2023-03-26 11:55:51,839 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=460 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/0a99a6b8255649f29d6389206f4fdb0d 2023-03-26 11:55:51,842 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/6b21eb239566489abcd6fd0ba1163e61 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/6b21eb239566489abcd6fd0ba1163e61 2023-03-26 11:55:51,844 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/6b21eb239566489abcd6fd0ba1163e61, entries=150, sequenceid=460, filesize=11.9 K 2023-03-26 11:55:51,844 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5fe385a08d874654967eb4993367e648 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5fe385a08d874654967eb4993367e648 2023-03-26 11:55:51,847 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5fe385a08d874654967eb4993367e648, entries=150, sequenceid=460, filesize=11.9 K 2023-03-26 11:55:51,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/0a99a6b8255649f29d6389206f4fdb0d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0a99a6b8255649f29d6389206f4fdb0d 2023-03-26 11:55:51,851 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0a99a6b8255649f29d6389206f4fdb0d, entries=150, sequenceid=460, filesize=11.9 K 2023-03-26 11:55:51,851 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=13.42 KB/13740 for e1f18589e0dd3700bbf1b6b539ec6e88 in 833ms, sequenceid=460, compaction requested=true 2023-03-26 11:55:51,851 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:51,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:51,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:51,851 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:51,851 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:51,851 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:51,852 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:51,852 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/A is initiating minor compaction (all files) 2023-03-26 11:55:51,852 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:51,852 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/A in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:51,852 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/B is initiating minor compaction (all files) 2023-03-26 11:55:51,853 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/607483c1203948089b40500aed22a904, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ddd1559c958f450f9cd7f6a08b91ab15, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/6b21eb239566489abcd6fd0ba1163e61] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.5 K 2023-03-26 11:55:51,853 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/B in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:51,853 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/151ab22567934ed3908167c3b864f681, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/7e17c474d7604f29aaeac50b499471d2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5fe385a08d874654967eb4993367e648] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.5 K 2023-03-26 11:55:51,853 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 607483c1203948089b40500aed22a904, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679831749179 2023-03-26 11:55:51,853 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 151ab22567934ed3908167c3b864f681, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679831749179 2023-03-26 11:55:51,853 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ddd1559c958f450f9cd7f6a08b91ab15, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1679831749259 2023-03-26 11:55:51,853 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 7e17c474d7604f29aaeac50b499471d2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1679831749259 2023-03-26 11:55:51,853 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 6b21eb239566489abcd6fd0ba1163e61, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679831750398 2023-03-26 11:55:51,853 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5fe385a08d874654967eb4993367e648, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679831750398 2023-03-26 11:55:51,860 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#A#compaction#492 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:51,860 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#B#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:52,273 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/5757e8de4e4c42f3a1f671a8140be021 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5757e8de4e4c42f3a1f671a8140be021 2023-03-26 11:55:52,273 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/3d7044301d684d55a298a45fffa73c7a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3d7044301d684d55a298a45fffa73c7a 2023-03-26 11:55:52,276 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/B of e1f18589e0dd3700bbf1b6b539ec6e88 into 5757e8de4e4c42f3a1f671a8140be021(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:52,276 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/A of e1f18589e0dd3700bbf1b6b539ec6e88 into 3d7044301d684d55a298a45fffa73c7a(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:52,276 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:52,276 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:52,276 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/B, priority=13, startTime=1679831751851; duration=0sec 2023-03-26 11:55:52,276 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/A, priority=13, startTime=1679831751851; duration=0sec 2023-03-26 11:55:52,277 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:55:52,277 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:52,277 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:55:52,277 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37342 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:55:52,277 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): e1f18589e0dd3700bbf1b6b539ec6e88/C is initiating minor compaction (all files) 2023-03-26 11:55:52,277 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of e1f18589e0dd3700bbf1b6b539ec6e88/C in TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:52,278 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/df553d06cc414045aa1adf78c8f195bd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6393e0c7babc491c97811547b07996c9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0a99a6b8255649f29d6389206f4fdb0d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp, totalSize=36.5 K 2023-03-26 11:55:52,278 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting df553d06cc414045aa1adf78c8f195bd, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=410, earliestPutTs=1679831749179 2023-03-26 11:55:52,278 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6393e0c7babc491c97811547b07996c9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1679831749259 2023-03-26 11:55:52,278 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 0a99a6b8255649f29d6389206f4fdb0d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=460, earliestPutTs=1679831750398 2023-03-26 11:55:52,283 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): e1f18589e0dd3700bbf1b6b539ec6e88#C#compaction#493 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:55:52,690 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/7d33679b7ccf4ddfb854f684e282086c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7d33679b7ccf4ddfb854f684e282086c 2023-03-26 11:55:52,694 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in e1f18589e0dd3700bbf1b6b539ec6e88/C of e1f18589e0dd3700bbf1b6b539ec6e88 into 7d33679b7ccf4ddfb854f684e282086c(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:55:52,694 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:52,694 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88., storeName=e1f18589e0dd3700bbf1b6b539ec6e88/C, priority=13, startTime=1679831751851; duration=0sec 2023-03-26 11:55:52,694 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:55:58,205 DEBUG [Thread-2993] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5726fb8b to 127.0.0.1:53955 2023-03-26 11:55:58,205 DEBUG [Thread-2993] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:58,309 DEBUG [Thread-2989] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x20ed9117 to 127.0.0.1:53955 2023-03-26 11:55:58,309 DEBUG [Thread-2989] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 94 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 100 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 2050 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 6146 rows 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 2056 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 6167 rows 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 2049 2023-03-26 11:55:58,310 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 6147 rows 2023-03-26 11:55:58,311 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 2056 2023-03-26 11:55:58,311 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 6168 rows 2023-03-26 11:55:58,311 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(402): scanned 2059 2023-03-26 11:55:58,311 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(403): verified 6177 rows 2023-03-26 11:55:58,311 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:55:58,311 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5ec943a9 to 127.0.0.1:53955 2023-03-26 11:55:58,311 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:55:58,314 INFO [Listener at localhost/46409] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 11:55:58,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 11:55:58,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 11:55:58,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 11:55:58,318 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831758318"}]},"ts":"1679831758318"} 2023-03-26 11:55:58,319 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 11:55:58,321 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 11:55:58,321 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, UNASSIGN}] 2023-03-26 11:55:58,322 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, UNASSIGN 2023-03-26 11:55:58,323 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=e1f18589e0dd3700bbf1b6b539ec6e88, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:55:58,323 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831758323"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831758323"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831758323"}]},"ts":"1679831758323"} 2023-03-26 11:55:58,324 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:55:58,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 11:55:58,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:58,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing e1f18589e0dd3700bbf1b6b539ec6e88, disabling compactions & flushes 2023-03-26 11:55:58,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:58,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:58,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. after waiting 0 ms 2023-03-26 11:55:58,475 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:58,475 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing e1f18589e0dd3700bbf1b6b539ec6e88 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 11:55:58,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=A 2023-03-26 11:55:58,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:58,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=B 2023-03-26 11:55:58,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:58,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK e1f18589e0dd3700bbf1b6b539ec6e88, store=C 2023-03-26 11:55:58,476 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:55:58,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 11:55:58,882 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/bdec3451373d46498a191b9ee634e2e3 2023-03-26 11:55:58,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 11:55:59,293 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/33223c5dda6f4f1087247712caa5d7ab 2023-03-26 11:55:59,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 11:55:59,701 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=470 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/f7a9ebe043704e198ca92062a36f8c4e 2023-03-26 11:55:59,704 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/A/bdec3451373d46498a191b9ee634e2e3 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bdec3451373d46498a191b9ee634e2e3 2023-03-26 11:55:59,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bdec3451373d46498a191b9ee634e2e3, entries=100, sequenceid=470, filesize=9.5 K 2023-03-26 11:55:59,720 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/B/33223c5dda6f4f1087247712caa5d7ab as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/33223c5dda6f4f1087247712caa5d7ab 2023-03-26 11:55:59,723 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/33223c5dda6f4f1087247712caa5d7ab, entries=100, sequenceid=470, filesize=9.5 K 2023-03-26 11:55:59,724 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/.tmp/C/f7a9ebe043704e198ca92062a36f8c4e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/f7a9ebe043704e198ca92062a36f8c4e 2023-03-26 11:55:59,728 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/f7a9ebe043704e198ca92062a36f8c4e, entries=100, sequenceid=470, filesize=9.5 K 2023-03-26 11:55:59,730 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for e1f18589e0dd3700bbf1b6b539ec6e88 in 1255ms, sequenceid=470, compaction requested=false 2023-03-26 11:55:59,768 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/e1154ac03a7541f8bfb481175112ac57, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/586fe521d1e24649ba9b19cbf4f6bbf5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/52669b68ffcf45548d312a328c97cde3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/d025ccb09bb848ae9bfecff8f88fcd05, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/81a130a1a025410f98900698ed59ffe8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/90617ece3b19443aaf3acf3f8a936aad, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ec9b3d9f44e648d48c09ed3fc620b9bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8f755ffa8091437aa2068586cfcabebb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/500040e83eb649c0a6c2e431647298de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b1b7c6df76c6471296699b188fb76bff, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3222e17fe89c403c892bc29904afe4e4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/fa0a86b0590145fc9be412da0fef4667, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/767a545909c74f059fe50894d75220d2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/eca05ea489094201b8f4d0d8956a2ac5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/c94af6b2b813406faecc6d006b88cce5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8e42e18ccedc4adf87afab1b1cbe4d37, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b9ca0404ae5048eebb6bb6b6e1669812, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a99febd4cac94f349ef7920e6040e681, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a54a33d1c38846d3925e5f2ebad52ee6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/61017bbc872342f0a17c6d719f139312, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/07944f23c79a477fa99898663a7bc7b0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/33967de1c74343e992a5a1a833085bbd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/0b5de64a20bd44b48061a33b868e9f8c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/607483c1203948089b40500aed22a904, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ddd1559c958f450f9cd7f6a08b91ab15, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/6b21eb239566489abcd6fd0ba1163e61] to archive 2023-03-26 11:55:59,769 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:59,770 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/e1154ac03a7541f8bfb481175112ac57 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/e1154ac03a7541f8bfb481175112ac57 2023-03-26 11:55:59,772 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/586fe521d1e24649ba9b19cbf4f6bbf5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/586fe521d1e24649ba9b19cbf4f6bbf5 2023-03-26 11:55:59,773 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/52669b68ffcf45548d312a328c97cde3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/52669b68ffcf45548d312a328c97cde3 2023-03-26 11:55:59,775 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/d025ccb09bb848ae9bfecff8f88fcd05 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/d025ccb09bb848ae9bfecff8f88fcd05 2023-03-26 11:55:59,776 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/81a130a1a025410f98900698ed59ffe8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/81a130a1a025410f98900698ed59ffe8 2023-03-26 11:55:59,777 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/90617ece3b19443aaf3acf3f8a936aad to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/90617ece3b19443aaf3acf3f8a936aad 2023-03-26 11:55:59,778 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ec9b3d9f44e648d48c09ed3fc620b9bf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ec9b3d9f44e648d48c09ed3fc620b9bf 2023-03-26 11:55:59,779 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8f755ffa8091437aa2068586cfcabebb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8f755ffa8091437aa2068586cfcabebb 2023-03-26 11:55:59,780 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/500040e83eb649c0a6c2e431647298de to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/500040e83eb649c0a6c2e431647298de 2023-03-26 11:55:59,782 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b1b7c6df76c6471296699b188fb76bff to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b1b7c6df76c6471296699b188fb76bff 2023-03-26 11:55:59,783 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3222e17fe89c403c892bc29904afe4e4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3222e17fe89c403c892bc29904afe4e4 2023-03-26 11:55:59,784 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/fa0a86b0590145fc9be412da0fef4667 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/fa0a86b0590145fc9be412da0fef4667 2023-03-26 11:55:59,785 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/767a545909c74f059fe50894d75220d2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/767a545909c74f059fe50894d75220d2 2023-03-26 11:55:59,786 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/eca05ea489094201b8f4d0d8956a2ac5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/eca05ea489094201b8f4d0d8956a2ac5 2023-03-26 11:55:59,787 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/c94af6b2b813406faecc6d006b88cce5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/c94af6b2b813406faecc6d006b88cce5 2023-03-26 11:55:59,788 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8e42e18ccedc4adf87afab1b1cbe4d37 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/8e42e18ccedc4adf87afab1b1cbe4d37 2023-03-26 11:55:59,789 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b9ca0404ae5048eebb6bb6b6e1669812 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/b9ca0404ae5048eebb6bb6b6e1669812 2023-03-26 11:55:59,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a99febd4cac94f349ef7920e6040e681 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a99febd4cac94f349ef7920e6040e681 2023-03-26 11:55:59,790 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a54a33d1c38846d3925e5f2ebad52ee6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/a54a33d1c38846d3925e5f2ebad52ee6 2023-03-26 11:55:59,791 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/61017bbc872342f0a17c6d719f139312 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/61017bbc872342f0a17c6d719f139312 2023-03-26 11:55:59,792 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/07944f23c79a477fa99898663a7bc7b0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/07944f23c79a477fa99898663a7bc7b0 2023-03-26 11:55:59,793 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/33967de1c74343e992a5a1a833085bbd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/33967de1c74343e992a5a1a833085bbd 2023-03-26 11:55:59,794 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/0b5de64a20bd44b48061a33b868e9f8c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/0b5de64a20bd44b48061a33b868e9f8c 2023-03-26 11:55:59,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/607483c1203948089b40500aed22a904 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/607483c1203948089b40500aed22a904 2023-03-26 11:55:59,795 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ddd1559c958f450f9cd7f6a08b91ab15 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/ddd1559c958f450f9cd7f6a08b91ab15 2023-03-26 11:55:59,796 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/6b21eb239566489abcd6fd0ba1163e61 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/6b21eb239566489abcd6fd0ba1163e61 2023-03-26 11:55:59,799 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/383307fcb33a49538a088a5d3f3242dd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/696858f039cb43ba941f89685eaac36e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e0b28e452bfa4fc89a5bf657296e8462, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/f08c8c05347c411a9938db1b51ba4218, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/8e32cfcdae1c4882b5ee9079c17953f9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6da68ac6b94b443abd2395c5c3fceae7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/b11d00589bf0455b9210f6b5e4e22d0c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e714f834bc7a4a43a8d29db205e16719, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5bb7c4ecbab14d1d811ac720b8e427b3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/84facc2ce281422081b4df00c578d440, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6d1c2a10359b486d8eef324cebc4c970, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2c95c9b6d64e40e7be35f5ea3d3bc462, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a6cd3bc31d604de3845ff3562e1848a2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e4218360a89d4516a6bbd9297f3a0f1d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/01fb599a07724cb281914273c3e6819b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/98385d9eb72a4f6784741738d5394578, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2f148b4f381749848573f8113203b46e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/384ce3f052f4403c8ab0726c55d78bcf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dcbfb8498a174c02939a0f61d1572bac, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/36a5b663cac24554848b1eb91002a800, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/afb75c0fa9b54e7f868d9a4dea280832, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5d01f750666643f097590f0e0275cbe9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/151ab22567934ed3908167c3b864f681, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6dce5d2bc4af4509bbaf21bf51877522, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/7e17c474d7604f29aaeac50b499471d2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5fe385a08d874654967eb4993367e648] to archive 2023-03-26 11:55:59,800 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:59,802 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/383307fcb33a49538a088a5d3f3242dd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/383307fcb33a49538a088a5d3f3242dd 2023-03-26 11:55:59,803 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/696858f039cb43ba941f89685eaac36e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/696858f039cb43ba941f89685eaac36e 2023-03-26 11:55:59,804 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e0b28e452bfa4fc89a5bf657296e8462 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e0b28e452bfa4fc89a5bf657296e8462 2023-03-26 11:55:59,805 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/f08c8c05347c411a9938db1b51ba4218 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/f08c8c05347c411a9938db1b51ba4218 2023-03-26 11:55:59,806 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/8e32cfcdae1c4882b5ee9079c17953f9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/8e32cfcdae1c4882b5ee9079c17953f9 2023-03-26 11:55:59,807 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6da68ac6b94b443abd2395c5c3fceae7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6da68ac6b94b443abd2395c5c3fceae7 2023-03-26 11:55:59,814 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/b11d00589bf0455b9210f6b5e4e22d0c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/b11d00589bf0455b9210f6b5e4e22d0c 2023-03-26 11:55:59,815 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e714f834bc7a4a43a8d29db205e16719 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e714f834bc7a4a43a8d29db205e16719 2023-03-26 11:55:59,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5bb7c4ecbab14d1d811ac720b8e427b3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5bb7c4ecbab14d1d811ac720b8e427b3 2023-03-26 11:55:59,816 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/84facc2ce281422081b4df00c578d440 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/84facc2ce281422081b4df00c578d440 2023-03-26 11:55:59,817 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6d1c2a10359b486d8eef324cebc4c970 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6d1c2a10359b486d8eef324cebc4c970 2023-03-26 11:55:59,818 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2c95c9b6d64e40e7be35f5ea3d3bc462 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2c95c9b6d64e40e7be35f5ea3d3bc462 2023-03-26 11:55:59,819 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a6cd3bc31d604de3845ff3562e1848a2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/a6cd3bc31d604de3845ff3562e1848a2 2023-03-26 11:55:59,820 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e4218360a89d4516a6bbd9297f3a0f1d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/e4218360a89d4516a6bbd9297f3a0f1d 2023-03-26 11:55:59,821 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/01fb599a07724cb281914273c3e6819b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/01fb599a07724cb281914273c3e6819b 2023-03-26 11:55:59,822 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/98385d9eb72a4f6784741738d5394578 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/98385d9eb72a4f6784741738d5394578 2023-03-26 11:55:59,824 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2f148b4f381749848573f8113203b46e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/2f148b4f381749848573f8113203b46e 2023-03-26 11:55:59,825 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/384ce3f052f4403c8ab0726c55d78bcf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/384ce3f052f4403c8ab0726c55d78bcf 2023-03-26 11:55:59,826 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dcbfb8498a174c02939a0f61d1572bac to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/dcbfb8498a174c02939a0f61d1572bac 2023-03-26 11:55:59,827 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/36a5b663cac24554848b1eb91002a800 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/36a5b663cac24554848b1eb91002a800 2023-03-26 11:55:59,828 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/afb75c0fa9b54e7f868d9a4dea280832 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/afb75c0fa9b54e7f868d9a4dea280832 2023-03-26 11:55:59,830 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5d01f750666643f097590f0e0275cbe9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5d01f750666643f097590f0e0275cbe9 2023-03-26 11:55:59,831 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/151ab22567934ed3908167c3b864f681 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/151ab22567934ed3908167c3b864f681 2023-03-26 11:55:59,832 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6dce5d2bc4af4509bbaf21bf51877522 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/6dce5d2bc4af4509bbaf21bf51877522 2023-03-26 11:55:59,833 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/7e17c474d7604f29aaeac50b499471d2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/7e17c474d7604f29aaeac50b499471d2 2023-03-26 11:55:59,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5fe385a08d874654967eb4993367e648 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5fe385a08d874654967eb4993367e648 2023-03-26 11:55:59,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b8e552cfd05a4cf98f7ec16a1caaebaf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/670db2ed6f0d44e58e36204b2b8009bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/bb72f3be41b4440fb9d844f34e08fedf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4860150ca2b741979e8e3b45e1688129, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5223938ea0fb4953830ab7f9c7a3685b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6299424c2b0b4b37b91d63b79000a84c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/217bda0b520c477ea95ba358b25eb838, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/dbb2b3defdfb4a6cb6731bed7d338a12, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/74b55e71f15246599ee10a1d10d191c2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/e11585c7790842898cf72d6ea3657653, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/9148885a96d74e51bcd69d51c687e323, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6333394780f94f589aa0be9590d7c344, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4a66b29660fd4607acf53af2f77e3698, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7701c04a52ea4cdb9c496eafb1b20211, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/aaf607974be1497b853e967e6c9209cc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5652a2b70fec451498ff2999a3616c6f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/31556b31214c4e349858ea9714d1b1a3, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/057259c67b624bf1bd3541222f159541, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/67b63939728a45228aae94d6ae2bfaa8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/de79ef26006b4975bf47bd05093c3122, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0feb93b31ffd42b2a03be5dc418de4f7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/fa822dc540c94a90a48245ff126a6b51, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/df553d06cc414045aa1adf78c8f195bd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3a908b3973f4497e912a1ad6ce745ca8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6393e0c7babc491c97811547b07996c9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0a99a6b8255649f29d6389206f4fdb0d] to archive 2023-03-26 11:55:59,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:55:59,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b8e552cfd05a4cf98f7ec16a1caaebaf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/b8e552cfd05a4cf98f7ec16a1caaebaf 2023-03-26 11:55:59,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/670db2ed6f0d44e58e36204b2b8009bf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/670db2ed6f0d44e58e36204b2b8009bf 2023-03-26 11:55:59,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/bb72f3be41b4440fb9d844f34e08fedf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/bb72f3be41b4440fb9d844f34e08fedf 2023-03-26 11:55:59,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4860150ca2b741979e8e3b45e1688129 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4860150ca2b741979e8e3b45e1688129 2023-03-26 11:55:59,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5223938ea0fb4953830ab7f9c7a3685b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5223938ea0fb4953830ab7f9c7a3685b 2023-03-26 11:55:59,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6299424c2b0b4b37b91d63b79000a84c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6299424c2b0b4b37b91d63b79000a84c 2023-03-26 11:55:59,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/217bda0b520c477ea95ba358b25eb838 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/217bda0b520c477ea95ba358b25eb838 2023-03-26 11:55:59,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/dbb2b3defdfb4a6cb6731bed7d338a12 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/dbb2b3defdfb4a6cb6731bed7d338a12 2023-03-26 11:55:59,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/74b55e71f15246599ee10a1d10d191c2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/74b55e71f15246599ee10a1d10d191c2 2023-03-26 11:55:59,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/e11585c7790842898cf72d6ea3657653 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/e11585c7790842898cf72d6ea3657653 2023-03-26 11:55:59,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/9148885a96d74e51bcd69d51c687e323 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/9148885a96d74e51bcd69d51c687e323 2023-03-26 11:55:59,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6333394780f94f589aa0be9590d7c344 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6333394780f94f589aa0be9590d7c344 2023-03-26 11:55:59,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4a66b29660fd4607acf53af2f77e3698 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/4a66b29660fd4607acf53af2f77e3698 2023-03-26 11:55:59,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7701c04a52ea4cdb9c496eafb1b20211 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7701c04a52ea4cdb9c496eafb1b20211 2023-03-26 11:55:59,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/aaf607974be1497b853e967e6c9209cc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/aaf607974be1497b853e967e6c9209cc 2023-03-26 11:55:59,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5652a2b70fec451498ff2999a3616c6f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/5652a2b70fec451498ff2999a3616c6f 2023-03-26 11:55:59,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/31556b31214c4e349858ea9714d1b1a3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/31556b31214c4e349858ea9714d1b1a3 2023-03-26 11:55:59,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/057259c67b624bf1bd3541222f159541 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/057259c67b624bf1bd3541222f159541 2023-03-26 11:55:59,861 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/67b63939728a45228aae94d6ae2bfaa8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/67b63939728a45228aae94d6ae2bfaa8 2023-03-26 11:55:59,862 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/de79ef26006b4975bf47bd05093c3122 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/de79ef26006b4975bf47bd05093c3122 2023-03-26 11:55:59,863 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0feb93b31ffd42b2a03be5dc418de4f7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0feb93b31ffd42b2a03be5dc418de4f7 2023-03-26 11:55:59,864 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/fa822dc540c94a90a48245ff126a6b51 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/fa822dc540c94a90a48245ff126a6b51 2023-03-26 11:55:59,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/df553d06cc414045aa1adf78c8f195bd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/df553d06cc414045aa1adf78c8f195bd 2023-03-26 11:55:59,867 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3a908b3973f4497e912a1ad6ce745ca8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/3a908b3973f4497e912a1ad6ce745ca8 2023-03-26 11:55:59,870 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6393e0c7babc491c97811547b07996c9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/6393e0c7babc491c97811547b07996c9 2023-03-26 11:55:59,871 DEBUG [StoreCloser-TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0a99a6b8255649f29d6389206f4fdb0d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/0a99a6b8255649f29d6389206f4fdb0d 2023-03-26 11:55:59,875 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/recovered.edits/473.seqid, newMaxSeqId=473, maxSeqId=1 2023-03-26 11:55:59,877 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88. 2023-03-26 11:55:59,877 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for e1f18589e0dd3700bbf1b6b539ec6e88: 2023-03-26 11:55:59,878 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:55:59,878 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=e1f18589e0dd3700bbf1b6b539ec6e88, regionState=CLOSED 2023-03-26 11:55:59,879 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831759878"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831759878"}]},"ts":"1679831759878"} 2023-03-26 11:55:59,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-03-26 11:55:59,881 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure e1f18589e0dd3700bbf1b6b539ec6e88, server=jenkins-hbase4.apache.org,38971,1679831615531 in 1.5560 sec 2023-03-26 11:55:59,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-03-26 11:55:59,882 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=e1f18589e0dd3700bbf1b6b539ec6e88, UNASSIGN in 1.5600 sec 2023-03-26 11:55:59,883 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831759883"}]},"ts":"1679831759883"} 2023-03-26 11:55:59,883 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 11:55:59,885 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 11:55:59,886 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5700 sec 2023-03-26 11:56:00,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-03-26 11:56:00,423 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-03-26 11:56:00,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 11:56:00,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,426 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-26 11:56:00,427 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,431 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:56:00,435 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/recovered.edits] 2023-03-26 11:56:00,438 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3d7044301d684d55a298a45fffa73c7a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/3d7044301d684d55a298a45fffa73c7a 2023-03-26 11:56:00,439 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bdec3451373d46498a191b9ee634e2e3 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/A/bdec3451373d46498a191b9ee634e2e3 2023-03-26 11:56:00,441 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/33223c5dda6f4f1087247712caa5d7ab to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/33223c5dda6f4f1087247712caa5d7ab 2023-03-26 11:56:00,442 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5757e8de4e4c42f3a1f671a8140be021 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/B/5757e8de4e4c42f3a1f671a8140be021 2023-03-26 11:56:00,444 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7d33679b7ccf4ddfb854f684e282086c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/7d33679b7ccf4ddfb854f684e282086c 2023-03-26 11:56:00,445 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/f7a9ebe043704e198ca92062a36f8c4e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/C/f7a9ebe043704e198ca92062a36f8c4e 2023-03-26 11:56:00,447 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/recovered.edits/473.seqid to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88/recovered.edits/473.seqid 2023-03-26 11:56:00,448 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/e1f18589e0dd3700bbf1b6b539ec6e88 2023-03-26 11:56:00,448 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:56:00,450 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,451 WARN [PEWorker-2] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 11:56:00,453 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 11:56:00,454 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,454 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 11:56:00,454 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679831760454"}]},"ts":"9223372036854775807"} 2023-03-26 11:56:00,455 INFO [PEWorker-2] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 11:56:00,455 DEBUG [PEWorker-2] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => e1f18589e0dd3700bbf1b6b539ec6e88, NAME => 'TestAcidGuarantees,,1679831728578.e1f18589e0dd3700bbf1b6b539ec6e88.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 11:56:00,455 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 11:56:00,456 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679831760455"}]},"ts":"9223372036854775807"} 2023-03-26 11:56:00,457 INFO [PEWorker-2] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 11:56:00,459 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,459 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 35 msec 2023-03-26 11:56:00,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-03-26 11:56:00,528 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-03-26 11:56:00,543 INFO [Listener at localhost/46409] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=225 (was 228), OpenFileDescriptor=554 (was 540) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=361 (was 346) - SystemLoadAverage LEAK? -, ProcessCount=169 (was 169), AvailableMemoryMB=9441 (was 8729) - AvailableMemoryMB LEAK? - 2023-03-26 11:56:00,558 INFO [Listener at localhost/46409] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225, OpenFileDescriptor=554, MaxFileDescriptor=60000, SystemLoadAverage=361, ProcessCount=169, AvailableMemoryMB=9440 2023-03-26 11:56:00,559 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:56:00,560 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:56:00,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:00,562 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-03-26 11:56:00,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-03-26 11:56:00,563 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-03-26 11:56:00,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 11:56:00,564 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:00,564 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 empty. 2023-03-26 11:56:00,565 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:00,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:56:00,572 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:56:00,574 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 5890b64a473c744d0cc3b74eead11b50, NAME => 'TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp 2023-03-26 11:56:00,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 11:56:00,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 11:56:00,982 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:56:00,982 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 5890b64a473c744d0cc3b74eead11b50, disabling compactions & flushes 2023-03-26 11:56:00,982 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:00,982 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:00,982 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. after waiting 0 ms 2023-03-26 11:56:00,982 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:00,982 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:00,982 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:00,985 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-03-26 11:56:00,985 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831760985"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831760985"}]},"ts":"1679831760985"} 2023-03-26 11:56:00,986 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-03-26 11:56:00,987 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-03-26 11:56:00,987 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831760987"}]},"ts":"1679831760987"} 2023-03-26 11:56:00,988 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-03-26 11:56:00,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, ASSIGN}] 2023-03-26 11:56:00,993 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, ASSIGN 2023-03-26 11:56:00,994 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=false 2023-03-26 11:56:01,145 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:01,146 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831761145"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831761145"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831761145"}]},"ts":"1679831761145"} 2023-03-26 11:56:01,147 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:56:01,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 11:56:01,303 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:01,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5890b64a473c744d0cc3b74eead11b50, NAME => 'TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:56:01,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:56:01,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,304 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,305 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,308 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A 2023-03-26 11:56:01,308 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A 2023-03-26 11:56:01,308 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:56:01,308 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5890b64a473c744d0cc3b74eead11b50 columnFamilyName A 2023-03-26 11:56:01,309 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(310): Store=5890b64a473c744d0cc3b74eead11b50/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:56:01,309 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,311 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B 2023-03-26 11:56:01,311 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B 2023-03-26 11:56:01,311 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:56:01,311 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5890b64a473c744d0cc3b74eead11b50 columnFamilyName B 2023-03-26 11:56:01,312 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(310): Store=5890b64a473c744d0cc3b74eead11b50/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:56:01,312 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,313 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C 2023-03-26 11:56:01,313 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C 2023-03-26 11:56:01,314 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:56:01,314 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5890b64a473c744d0cc3b74eead11b50 columnFamilyName C 2023-03-26 11:56:01,314 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(310): Store=5890b64a473c744d0cc3b74eead11b50/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:56:01,315 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:01,315 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,316 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,318 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:56:01,319 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:01,321 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-03-26 11:56:01,322 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5890b64a473c744d0cc3b74eead11b50; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=67486567, jitterRate=0.005628213286399841}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:56:01,322 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:01,323 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., pid=56, masterSystemTime=1679831761299 2023-03-26 11:56:01,324 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:01,324 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:01,325 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:01,325 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831761325"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831761325"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831761325"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831761325"}]},"ts":"1679831761325"} 2023-03-26 11:56:01,327 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-03-26 11:56:01,327 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 in 179 msec 2023-03-26 11:56:01,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-03-26 11:56:01,329 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, ASSIGN in 335 msec 2023-03-26 11:56:01,329 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-03-26 11:56:01,330 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831761330"}]},"ts":"1679831761330"} 2023-03-26 11:56:01,330 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-03-26 11:56:01,333 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-03-26 11:56:01,334 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 773 msec 2023-03-26 11:56:01,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-03-26 11:56:01,667 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-03-26 11:56:01,668 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x03ced4ba to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:01,673 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@45878790, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:01,675 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:01,676 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35726, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:01,677 DEBUG [Listener at localhost/46409] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-03-26 11:56:01,680 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50464, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-03-26 11:56:01,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-03-26 11:56:01,681 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-03-26 11:56:01,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:02,103 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 11:56:02,105 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-03-26 11:56:02,105 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-03-26 11:56:02,108 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-03-26 11:56:02,110 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, REOPEN/MOVE}] 2023-03-26 11:56:02,111 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, REOPEN/MOVE 2023-03-26 11:56:02,112 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,112 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831762112"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831762112"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831762112"}]},"ts":"1679831762112"} 2023-03-26 11:56:02,116 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:56:02,268 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5890b64a473c744d0cc3b74eead11b50, disabling compactions & flushes 2023-03-26 11:56:02,269 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. after waiting 0 ms 2023-03-26 11:56:02,269 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,273 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-03-26 11:56:02,274 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,274 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:02,274 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 5890b64a473c744d0cc3b74eead11b50 to self. 2023-03-26 11:56:02,275 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,276 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=CLOSED 2023-03-26 11:56:02,276 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831762276"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831762276"}]},"ts":"1679831762276"} 2023-03-26 11:56:02,278 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-03-26 11:56:02,278 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 in 161 msec 2023-03-26 11:56:02,279 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,38971,1679831615531; forceNewPlan=false, retain=true 2023-03-26 11:56:02,429 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,430 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831762429"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831762429"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831762429"}]},"ts":"1679831762429"} 2023-03-26 11:56:02,431 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:56:02,586 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 5890b64a473c744d0cc3b74eead11b50, NAME => 'TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.', STARTKEY => '', ENDKEY => ''} 2023-03-26 11:56:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-03-26 11:56:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,587 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,588 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,589 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A 2023-03-26 11:56:02,589 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A 2023-03-26 11:56:02,590 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:56:02,590 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5890b64a473c744d0cc3b74eead11b50 columnFamilyName A 2023-03-26 11:56:02,591 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(310): Store=5890b64a473c744d0cc3b74eead11b50/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:56:02,592 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,592 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B 2023-03-26 11:56:02,593 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B 2023-03-26 11:56:02,593 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:56:02,593 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5890b64a473c744d0cc3b74eead11b50 columnFamilyName B 2023-03-26 11:56:02,593 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(310): Store=5890b64a473c744d0cc3b74eead11b50/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:56:02,593 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,594 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C 2023-03-26 11:56:02,594 DEBUG [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C 2023-03-26 11:56:02,594 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-03-26 11:56:02,594 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 5890b64a473c744d0cc3b74eead11b50 columnFamilyName C 2023-03-26 11:56:02,595 INFO [StoreOpener-5890b64a473c744d0cc3b74eead11b50-1] regionserver.HStore(310): Store=5890b64a473c744d0cc3b74eead11b50/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-03-26 11:56:02,595 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,596 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,597 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,598 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-03-26 11:56:02,599 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,600 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 5890b64a473c744d0cc3b74eead11b50; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=74083259, jitterRate=0.10392658412456512}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-03-26 11:56:02,600 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:02,601 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., pid=61, masterSystemTime=1679831762583 2023-03-26 11:56:02,602 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,602 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,602 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,603 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831762602"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1679831762602"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1679831762602"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1679831762602"}]},"ts":"1679831762602"} 2023-03-26 11:56:02,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-03-26 11:56:02,605 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 in 172 msec 2023-03-26 11:56:02,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-03-26 11:56:02,606 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, REOPEN/MOVE in 495 msec 2023-03-26 11:56:02,608 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-03-26 11:56:02,608 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 499 msec 2023-03-26 11:56:02,610 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 927 msec 2023-03-26 11:56:02,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-03-26 11:56:02,611 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x63da10ca to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,617 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1644730f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,618 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x59185ba7 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,623 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6af74c0e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,624 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x062b3caa to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,629 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1d759aab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,629 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x66b5b1f3 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,635 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@39241fe3, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,635 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x544a4215 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,642 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1fbb5b46, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,643 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x23e65a76 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,652 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@66c25427, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,652 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x5e6a6f3c to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,659 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4f4eab6d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,660 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x0a470a3d to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,672 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7d997c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,673 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x4bb72437 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,679 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7993d99e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,680 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(139): Connect 0x3dca23c0 to 127.0.0.1:53955 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-03-26 11:56:02,690 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fae3ed2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-03-26 11:56:02,720 DEBUG [hconnection-0x636d1bce-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,724 DEBUG [hconnection-0x6a3f81e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,725 DEBUG [hconnection-0x239ea569-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,732 DEBUG [hconnection-0xa0357c8-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,733 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35754, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,733 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35742, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,734 DEBUG [hconnection-0x54868703-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,734 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35762, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,736 DEBUG [hconnection-0x5993f6b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,737 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35794, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:02,740 DEBUG [hconnection-0x47cf1588-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,742 DEBUG [hconnection-0x57dd05a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,744 DEBUG [hconnection-0x59c83892-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,745 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35824, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,750 DEBUG [hconnection-0x7fe5a5f3-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-03-26 11:56:02,755 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35772, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,755 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35780, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,755 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35802, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,755 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35814, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,755 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35832, version=2.4.17-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-03-26 11:56:02,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:02,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:02,767 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:56:02,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:02,767 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:02,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:02,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:02,767 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:02,767 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:02,767 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:02,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:02,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:02,768 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:02,768 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:02,769 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,769 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,771 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:02,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:02,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:02,771 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:02,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:02,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,774 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:02,781 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:02,781 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:02,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:02,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:02,781 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:02,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:02,784 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:02,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:02,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:02,785 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:02,785 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:02,787 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:02,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:02,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:02,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:02,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48d8388a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:02,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:02,789 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,790 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:02,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:02,790 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:02,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,791 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831822787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831822789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,792 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:02,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:02,792 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:02,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:02,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831822790, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,794 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,795 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831822791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831822792, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:02,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:02,796 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:02,796 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:02,796 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:02,796 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:02,796 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:02,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:02,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:02,802 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,803 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:02,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:02,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:02,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:02,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:02,814 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,814 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,814 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:02,814 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:02,815 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:02,815 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:02,815 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:02,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:02,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:02,827 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326306de39e66174d0994a3688a5373ebdf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326306de39e66174d0994a3688a5373ebdf 2023-03-26 11:56:02,836 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/ef0e16e0673345a697c026fb8f1d3663 2023-03-26 11:56:02,890 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/c3d526f9a466439dbb3408008a25283a 2023-03-26 11:56:02,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831822893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831822893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831822896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831822896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:02,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831822896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,915 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:02,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:02,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:02,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:02,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:02,932 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:02,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:02,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:02,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:02,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,933 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,946 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:02,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:02,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:02,946 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,947 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,947 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:02,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:02,952 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:02,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:02,960 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:02,960 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,960 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:02,961 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:02,961 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:02,961 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:02,963 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,963 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,964 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@295608f6[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:02,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:02,965 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,966 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:02,966 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:02,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,967 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:02,968 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:02,969 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:02,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:02,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:02,971 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:02,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:02,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:02,972 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:02,972 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:02,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:02,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:02,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:02,976 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:02,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,979 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:02,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:02,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:02,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:02,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:02,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:02,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:02,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:02,991 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:02,991 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:02,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:02,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:02,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,091 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,094 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831823096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,098 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,098 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831823097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,100 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,100 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,101 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,101 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831823099, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831823100, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,101 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831823101, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,102 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,102 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,103 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,103 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,103 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,108 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,108 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,108 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:03,120 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,120 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:03,120 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:03,120 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:03,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:03,122 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,122 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,123 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:03,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d93427c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:03,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:03,124 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,127 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:03,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:03,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,127 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:03,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:03,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:03,132 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:03,132 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:03,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:03,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:03,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:03,132 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:03,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,134 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,134 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,136 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,138 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:03,139 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:03,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,139 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:03,139 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:03,139 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:03,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,239 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,243 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,243 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,244 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,244 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,246 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,247 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:03,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:03,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:03,249 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:03,251 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,251 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5688d9ca[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:03,254 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:03,254 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,255 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,255 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:03,255 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:03,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:03,256 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:03,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,256 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,257 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:03,260 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:03,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:03,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:03,260 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:03,260 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:03,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,264 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,267 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,269 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,269 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,269 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,269 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,270 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,270 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,270 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,270 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,270 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,270 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:03,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:03,270 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:03,270 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:03,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:03,323 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/fcfd46e696bc4e44890cc25cd51e8ee8 2023-03-26 11:56:03,327 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/ef0e16e0673345a697c026fb8f1d3663 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ef0e16e0673345a697c026fb8f1d3663 2023-03-26 11:56:03,331 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ef0e16e0673345a697c026fb8f1d3663, entries=150, sequenceid=16, filesize=25.1 K 2023-03-26 11:56:03,332 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/c3d526f9a466439dbb3408008a25283a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3d526f9a466439dbb3408008a25283a 2023-03-26 11:56:03,335 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3d526f9a466439dbb3408008a25283a, entries=150, sequenceid=16, filesize=11.6 K 2023-03-26 11:56:03,336 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/fcfd46e696bc4e44890cc25cd51e8ee8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fcfd46e696bc4e44890cc25cd51e8ee8 2023-03-26 11:56:03,339 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fcfd46e696bc4e44890cc25cd51e8ee8, entries=150, sequenceid=16, filesize=11.6 K 2023-03-26 11:56:03,340 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 5890b64a473c744d0cc3b74eead11b50 in 573ms, sequenceid=16, compaction requested=false 2023-03-26 11:56:03,340 DEBUG [MemStoreFlusher.0] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-03-26 11:56:03,341 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:03,370 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,373 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,380 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,381 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,381 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,381 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,382 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,382 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:03,382 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:03,382 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,382 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:03,383 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:56:03,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:03,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:03,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:03,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:03,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:03,383 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:03,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:03,400 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:03,410 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e6701e156cef4e76bc6bc6fcb3b93763 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e6701e156cef4e76bc6bc6fcb3b93763 2023-03-26 11:56:03,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831823408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831823408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831823408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,414 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831823411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,416 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831823413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,416 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=36, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/76d02416f6bf41c9a8e25214b63c83b2 2023-03-26 11:56:03,427 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/5b52c0f5215e43db869233c3d21dc686 2023-03-26 11:56:03,456 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=36 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/e88f54432e674fdfb523eb5d26135ceb 2023-03-26 11:56:03,461 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/76d02416f6bf41c9a8e25214b63c83b2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/76d02416f6bf41c9a8e25214b63c83b2 2023-03-26 11:56:03,464 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/76d02416f6bf41c9a8e25214b63c83b2, entries=150, sequenceid=36, filesize=25.1 K 2023-03-26 11:56:03,465 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/5b52c0f5215e43db869233c3d21dc686 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5b52c0f5215e43db869233c3d21dc686 2023-03-26 11:56:03,469 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5b52c0f5215e43db869233c3d21dc686, entries=150, sequenceid=36, filesize=11.6 K 2023-03-26 11:56:03,470 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/e88f54432e674fdfb523eb5d26135ceb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/e88f54432e674fdfb523eb5d26135ceb 2023-03-26 11:56:03,474 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/e88f54432e674fdfb523eb5d26135ceb, entries=150, sequenceid=36, filesize=11.6 K 2023-03-26 11:56:03,474 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 5890b64a473c744d0cc3b74eead11b50 in 91ms, sequenceid=36, compaction requested=false 2023-03-26 11:56:03,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:03,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:03,476 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:03,479 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:03,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f1db7f9[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:03,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:03,481 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,482 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:03,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:03,483 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:03,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:03,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:03,485 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:03,488 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:03,488 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:03,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:03,489 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:03,489 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:03,489 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:03,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,491 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:03,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:03,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,500 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,500 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:03,500 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:03,500 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:03,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:03,517 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-03-26 11:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:03,518 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:03,530 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326cdb9cfe0591b4c4f999d3ffcfe1dfbec to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326cdb9cfe0591b4c4f999d3ffcfe1dfbec 2023-03-26 11:56:03,533 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831823528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,537 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,537 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831823532, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,539 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831823533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,540 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831823533, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,544 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831823542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,600 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,603 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,605 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,606 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:03,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:03,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:03,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:03,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,609 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,609 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:03,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:03,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46951554[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:03,611 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:03,611 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,612 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:03,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:03,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:03,612 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,612 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:03,614 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:03,617 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:03,617 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:03,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:03,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:03,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:03,617 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:03,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,619 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,619 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:03,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:03,623 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,623 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:03,623 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:03,624 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,624 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:03,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831823634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,640 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831823638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831823640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831823641, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831823645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,724 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,726 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,727 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,727 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,727 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,727 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,727 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,728 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,728 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,729 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,730 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,730 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,731 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:03,731 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:03,732 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,732 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:03,732 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:03,732 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:03,732 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:03,735 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:03,737 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41122f58[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:03,737 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:03,737 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:03,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:03,738 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:03,738 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,738 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:03,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:03,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,740 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:03,743 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:03,743 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:03,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:03,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:03,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:03,743 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:03,744 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,744 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,745 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,745 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,747 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,750 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:03,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:03,750 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,751 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:03,751 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:03,839 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831823838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,844 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831823842, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,846 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831823844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831823846, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,848 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:03,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831823847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,851 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,853 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,854 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,854 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,855 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,856 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,857 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,857 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,858 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,858 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:03,858 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,858 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:03,858 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:03,858 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:03,858 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:03,863 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,863 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,863 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,863 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:03,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:03,865 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@51c4f839[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:03,865 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:03,865 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:03,868 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:03,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:03,868 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,869 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:03,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:03,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:03,872 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:03,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:03,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:03,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:03,872 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:03,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:03,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:03,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:03,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:03,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:03,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:03,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,880 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:03,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:03,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:03,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,880 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:03,880 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:03,880 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:03,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:03,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:03,942 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=52, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/980c0e8978314f15a8c0914de0ed3adc 2023-03-26 11:56:03,953 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/e2a40fcd3deb4d66a678a6ebc487ab29 2023-03-26 11:56:03,980 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=52 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/39238d8380664723b1681c735d072c01 2023-03-26 11:56:03,980 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:03,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:03,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:03,985 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/980c0e8978314f15a8c0914de0ed3adc as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/980c0e8978314f15a8c0914de0ed3adc 2023-03-26 11:56:03,990 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/980c0e8978314f15a8c0914de0ed3adc, entries=200, sequenceid=52, filesize=31.9 K 2023-03-26 11:56:03,991 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/e2a40fcd3deb4d66a678a6ebc487ab29 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e2a40fcd3deb4d66a678a6ebc487ab29 2023-03-26 11:56:03,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:03,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:03,992 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:03,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:03,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:03,992 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,995 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:03,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:03,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:03,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:03,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:03,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:03,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:03,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:03,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:03,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:03,997 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:03,997 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e2a40fcd3deb4d66a678a6ebc487ab29, entries=150, sequenceid=52, filesize=11.6 K 2023-03-26 11:56:03,998 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/39238d8380664723b1681c735d072c01 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/39238d8380664723b1681c735d072c01 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,001 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,002 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/39238d8380664723b1681c735d072c01, entries=150, sequenceid=52, filesize=11.6 K 2023-03-26 11:56:04,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 5890b64a473c744d0cc3b74eead11b50 in 486ms, sequenceid=52, compaction requested=true 2023-03-26 11:56:04,003 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,003 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:04,003 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:04,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,004 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:04,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:04,004 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:04,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,005 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:04,005 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84178 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:04,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,005 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:04,005 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:04,005 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,005 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,005 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3d526f9a466439dbb3408008a25283a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5b52c0f5215e43db869233c3d21dc686, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e2a40fcd3deb4d66a678a6ebc487ab29] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=34.8 K 2023-03-26 11:56:04,005 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ef0e16e0673345a697c026fb8f1d3663, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/76d02416f6bf41c9a8e25214b63c83b2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/980c0e8978314f15a8c0914de0ed3adc] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=82.2 K 2023-03-26 11:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,006 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63d42f87[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,006 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c3d526f9a466439dbb3408008a25283a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679831762760 2023-03-26 11:56:04,006 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ef0e16e0673345a697c026fb8f1d3663, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679831762760 2023-03-26 11:56:04,007 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5b52c0f5215e43db869233c3d21dc686, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679831762784 2023-03-26 11:56:04,007 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 76d02416f6bf41c9a8e25214b63c83b2, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679831762784 2023-03-26 11:56:04,007 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 980c0e8978314f15a8c0914de0ed3adc, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679831763411 2023-03-26 11:56:04,007 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e2a40fcd3deb4d66a678a6ebc487ab29, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679831763515 2023-03-26 11:56:04,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,011 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,015 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#506 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,017 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,017 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,017 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,017 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,017 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,022 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,025 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,025 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,025 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:04,041 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#507 average throughput is 0.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:04,126 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:04,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:04,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:04,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:04,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:04,129 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:04,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:04,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:04,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,131 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:04,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:04,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:04,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:04,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,133 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,133 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:04,133 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:04,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:04,144 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,158 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831824155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831824156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831824156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,161 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326932768420e344784a3a7cc8b3cc007a2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326932768420e344784a3a7cc8b3cc007a2 2023-03-26 11:56:04,161 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831824157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831824158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,174 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=69, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/3b7e40145d554ce89b2443699c2d26c6 2023-03-26 11:56:04,194 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/3741a6213a0142509f129c71b3ef1e8e 2023-03-26 11:56:04,205 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=69 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/5b0e139a2dfb4fd39f4c432ef6c0443b 2023-03-26 11:56:04,208 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/3b7e40145d554ce89b2443699c2d26c6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b7e40145d554ce89b2443699c2d26c6 2023-03-26 11:56:04,211 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b7e40145d554ce89b2443699c2d26c6, entries=150, sequenceid=69, filesize=25.1 K 2023-03-26 11:56:04,212 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/3741a6213a0142509f129c71b3ef1e8e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/3741a6213a0142509f129c71b3ef1e8e 2023-03-26 11:56:04,215 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/3741a6213a0142509f129c71b3ef1e8e, entries=150, sequenceid=69, filesize=11.6 K 2023-03-26 11:56:04,216 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/5b0e139a2dfb4fd39f4c432ef6c0443b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/5b0e139a2dfb4fd39f4c432ef6c0443b 2023-03-26 11:56:04,218 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/5b0e139a2dfb4fd39f4c432ef6c0443b, entries=150, sequenceid=69, filesize=11.6 K 2023-03-26 11:56:04,219 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5890b64a473c744d0cc3b74eead11b50 in 86ms, sequenceid=69, compaction requested=true 2023-03-26 11:56:04,219 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:04,219 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,219 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,220 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,222 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,222 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,224 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d85b91c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,224 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,225 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,226 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,226 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,227 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,227 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,230 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,230 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,230 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,230 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,230 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,232 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,233 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,233 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,233 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,237 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:04,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,238 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,238 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,238 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,238 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,238 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,238 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:04,261 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:56:04,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:04,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:04,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:04,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:04,262 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:04,262 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:04,277 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267f0e10315727486e9b4a6ab5f218000a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267f0e10315727486e9b4a6ab5f218000a 2023-03-26 11:56:04,278 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831824273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831824277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831824278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831824278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,282 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831824279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,338 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:04,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:04,340 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:04,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:04,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:04,341 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:04,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:04,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:04,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,342 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,343 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,343 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,343 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,344 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:04,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,344 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,345 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:04,345 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:04,345 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:04,345 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:04,345 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,345 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:04,345 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,346 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,346 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,346 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,346 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,346 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,347 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,347 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,348 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,350 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5dbd2f1b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,350 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,353 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,353 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,353 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,353 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,353 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,353 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,357 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,358 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,360 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,360 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,360 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,360 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,360 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,362 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,363 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,365 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,369 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,369 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,370 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,370 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,376 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:04,381 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831824379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831824382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831824383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831824383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,386 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831824384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,456 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/2be66a55356643f59141f8f91ad801ea as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2be66a55356643f59141f8f91ad801ea 2023-03-26 11:56:04,464 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 2be66a55356643f59141f8f91ad801ea(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:04,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:04,464 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831764004; duration=0sec 2023-03-26 11:56:04,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:04,464 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:04,465 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:04,465 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:04,466 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,466 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fcfd46e696bc4e44890cc25cd51e8ee8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/e88f54432e674fdfb523eb5d26135ceb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/39238d8380664723b1681c735d072c01, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/5b0e139a2dfb4fd39f4c432ef6c0443b] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=46.4 K 2023-03-26 11:56:04,466 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fcfd46e696bc4e44890cc25cd51e8ee8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1679831762760 2023-03-26 11:56:04,466 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e88f54432e674fdfb523eb5d26135ceb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=36, earliestPutTs=1679831762784 2023-03-26 11:56:04,466 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 39238d8380664723b1681c735d072c01, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679831763515 2023-03-26 11:56:04,467 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5b0e139a2dfb4fd39f4c432ef6c0443b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679831763530 2023-03-26 11:56:04,470 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:04,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:04,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:04,474 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#512 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:04,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:04,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:04,475 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:04,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:04,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:04,476 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,476 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,477 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:04,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:04,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:04,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:04,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:04,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,491 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,491 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,492 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,492 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,492 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,494 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,494 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,494 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bce01a5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,497 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,502 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,502 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,504 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/4c973dfc185d4d2e8ecf0ae7e6222893 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/4c973dfc185d4d2e8ecf0ae7e6222893 2023-03-26 11:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,505 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,505 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,505 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,507 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,508 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,510 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,510 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 4c973dfc185d4d2e8ecf0ae7e6222893(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:04,510 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:04,510 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=12, startTime=1679831764004; duration=0sec 2023-03-26 11:56:04,510 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,513 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,513 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,513 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,513 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:04,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831824582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,588 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831824585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831824586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,589 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831824586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,590 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831824588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,613 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:04,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:04,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:04,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:04,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:04,617 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:04,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:04,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:04,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,617 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,618 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,618 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:04,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:04,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,623 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,624 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,625 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6924ab4c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,626 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,626 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,626 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,627 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,630 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,630 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,630 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,632 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,632 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,632 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,632 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,632 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,634 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,637 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,637 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,637 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:04,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,686 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=85, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/da0b1a3a95fb4b40bd8f9b032dd2798d 2023-03-26 11:56:04,737 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:04,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:04,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:04,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:04,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:04,741 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:04,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:04,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:04,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:04,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,744 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,745 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:04,745 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:04,745 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:04,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:04,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:04,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,745 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,745 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,745 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,745 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,746 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,747 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49c9a22d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,750 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,750 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,751 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,752 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,752 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,754 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,754 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,754 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,754 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,762 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,762 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,762 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,763 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,773 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,776 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,776 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,776 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,777 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,777 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,877 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:04,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:04,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:04,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:04,880 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:04,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:04,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:04,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,884 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:04,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:04,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,884 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:04,890 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:04,890 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:04,890 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,890 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:04,890 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:04,890 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:04,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:04,890 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:04,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831824888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:04,891 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:04,892 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831824890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,892 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:04,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d3899f8[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:04,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:04,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831824891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,894 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,895 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:04,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:04,896 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,896 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:04,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831824892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:04,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831824893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,897 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,898 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:04,900 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:04,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:04,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:04,900 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:04,900 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:04,902 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,902 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:04,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,902 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,903 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,903 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:04,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,906 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:04,907 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:04,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:04,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:04,907 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:04,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:04,907 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:04,908 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:04,908 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:04,908 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:04,926 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/269d925e977f4a60b5733de65e995713 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/269d925e977f4a60b5733de65e995713 2023-03-26 11:56:04,931 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 269d925e977f4a60b5733de65e995713(size=25.2 K), total size for store is 50.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:04,931 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:04,931 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831764003; duration=0sec 2023-03-26 11:56:04,931 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:05,008 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:05,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:05,010 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:05,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:05,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:05,011 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:05,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:05,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:05,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:05,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,013 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:05,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:05,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:05,014 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,014 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:05,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,014 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:05,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,014 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:05,015 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:05,015 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:05,015 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,015 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:05,015 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:05,015 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:05,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:05,017 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,017 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:05,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:05,018 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,018 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:05,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:05,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:05,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:05,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:05,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26207cd3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:05,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:05,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,022 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:05,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:05,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:05,022 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,022 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:05,024 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,024 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:05,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:05,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:05,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:05,026 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:05,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:05,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:05,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:05,027 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:05,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:05,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:05,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:05,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,030 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:05,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:05,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:05,033 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:05,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:05,033 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:05,033 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:05,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:05,032 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:05,032 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,033 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,033 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,033 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:05,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:05,033 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,098 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/13adf8e150c74b85920458acaea5e02d 2023-03-26 11:56:05,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=85 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/919bf69243124aa7a7bb80d426b355c4 2023-03-26 11:56:05,112 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/da0b1a3a95fb4b40bd8f9b032dd2798d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/da0b1a3a95fb4b40bd8f9b032dd2798d 2023-03-26 11:56:05,116 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/da0b1a3a95fb4b40bd8f9b032dd2798d, entries=200, sequenceid=85, filesize=31.9 K 2023-03-26 11:56:05,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/13adf8e150c74b85920458acaea5e02d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13adf8e150c74b85920458acaea5e02d 2023-03-26 11:56:05,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13adf8e150c74b85920458acaea5e02d, entries=150, sequenceid=85, filesize=11.6 K 2023-03-26 11:56:05,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/919bf69243124aa7a7bb80d426b355c4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/919bf69243124aa7a7bb80d426b355c4 2023-03-26 11:56:05,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/919bf69243124aa7a7bb80d426b355c4, entries=150, sequenceid=85, filesize=11.6 K 2023-03-26 11:56:05,124 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 5890b64a473c744d0cc3b74eead11b50 in 863ms, sequenceid=85, compaction requested=true 2023-03-26 11:56:05,125 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:05,125 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:05,125 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:05,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:05,125 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:05,126 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:05,126 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:05,126 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,126 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2be66a55356643f59141f8f91ad801ea, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/3741a6213a0142509f129c71b3ef1e8e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13adf8e150c74b85920458acaea5e02d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=34.9 K 2023-03-26 11:56:05,127 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84281 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:05,127 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:05,127 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,127 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/269d925e977f4a60b5733de65e995713, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b7e40145d554ce89b2443699c2d26c6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/da0b1a3a95fb4b40bd8f9b032dd2798d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=82.3 K 2023-03-26 11:56:05,127 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 269d925e977f4a60b5733de65e995713, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679831763515 2023-03-26 11:56:05,128 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2be66a55356643f59141f8f91ad801ea, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=52, earliestPutTs=1679831763515 2023-03-26 11:56:05,128 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3b7e40145d554ce89b2443699c2d26c6, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679831763530 2023-03-26 11:56:05,128 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3741a6213a0142509f129c71b3ef1e8e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679831763530 2023-03-26 11:56:05,128 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting da0b1a3a95fb4b40bd8f9b032dd2798d, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679831764152 2023-03-26 11:56:05,129 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 13adf8e150c74b85920458acaea5e02d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679831764155 2023-03-26 11:56:05,133 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:05,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:05,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:05,137 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#515 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:05,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:05,139 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:05,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:05,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:05,139 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:05,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,142 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:05,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:05,142 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,142 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,142 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:05,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:05,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,143 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:05,144 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:05,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:05,145 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:05,145 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:05,145 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:05,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:05,164 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:05,164 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 11:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:05,164 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:05,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:05,165 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,166 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#516 average throughput is 0.73 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:05,181 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/a2dd2c5b36e94ac6b7b608bd6f6c42f2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a2dd2c5b36e94ac6b7b608bd6f6c42f2 2023-03-26 11:56:05,194 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into a2dd2c5b36e94ac6b7b608bd6f6c42f2(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:05,194 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,195 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831765125; duration=0sec 2023-03-26 11:56:05,195 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:05,195 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-03-26 11:56:05,196 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 11:56:05,196 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 11:56:05,196 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. because compaction request was cancelled 2023-03-26 11:56:05,198 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326c67167a58d074d95b549c1ef92431eed to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c67167a58d074d95b549c1ef92431eed 2023-03-26 11:56:05,226 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=105, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/96ff62b902f34b3dae9155e469aba053 2023-03-26 11:56:05,230 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/a66104a2c84e4700bbc346ca5e2c17b5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a66104a2c84e4700bbc346ca5e2c17b5 2023-03-26 11:56:05,234 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into a66104a2c84e4700bbc346ca5e2c17b5(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:05,234 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,234 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831765125; duration=0sec 2023-03-26 11:56:05,234 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:05,246 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/1965077d684d4082b064054d4dc2e125 2023-03-26 11:56:05,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:05,395 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:05,405 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831825403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831825404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,421 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831825404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,422 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831825405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831825405, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,422 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-03-26 11:56:05,508 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831825506, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831825523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831825523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,524 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831825523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831825523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,655 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=105 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/a896df2ade074958b64fac5762d6dbc2 2023-03-26 11:56:05,659 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/96ff62b902f34b3dae9155e469aba053 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96ff62b902f34b3dae9155e469aba053 2023-03-26 11:56:05,663 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96ff62b902f34b3dae9155e469aba053, entries=150, sequenceid=105, filesize=25.1 K 2023-03-26 11:56:05,663 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/1965077d684d4082b064054d4dc2e125 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/1965077d684d4082b064054d4dc2e125 2023-03-26 11:56:05,667 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/1965077d684d4082b064054d4dc2e125, entries=150, sequenceid=105, filesize=11.6 K 2023-03-26 11:56:05,668 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/a896df2ade074958b64fac5762d6dbc2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a896df2ade074958b64fac5762d6dbc2 2023-03-26 11:56:05,671 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a896df2ade074958b64fac5762d6dbc2, entries=150, sequenceid=105, filesize=11.6 K 2023-03-26 11:56:05,671 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5890b64a473c744d0cc3b74eead11b50 in 507ms, sequenceid=105, compaction requested=true 2023-03-26 11:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,671 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,671 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:05,672 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:05,672 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,672 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:05,672 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:05,673 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,673 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:05,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:05,674 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:05,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:05,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:05,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:05,678 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a92dd90[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:05,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:05,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,679 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:05,679 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:05,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:05,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:05,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:05,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:05,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:05,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:05,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:05,685 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:05,685 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:05,685 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:05,685 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:05,685 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:05,688 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,688 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:05,688 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:05,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:05,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,692 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:05,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:05,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:05,698 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:05,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:05,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:05,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,698 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:05,698 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:05,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:05,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:05,698 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,699 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:05,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:05,712 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:56:05,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:05,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:05,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,712 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:05,712 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,729 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303269091a5bbe1c740cbb3c258177aac5599 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269091a5bbe1c740cbb3c258177aac5599 2023-03-26 11:56:05,733 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=123, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/8f50bc98398d45f7be237f2379ab2437 2023-03-26 11:56:05,734 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831825731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,735 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,735 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831825732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831825732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831825733, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,737 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831825734, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,742 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/a7e6bb3552614e7284b84c095117909a 2023-03-26 11:56:05,753 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=123 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/b5b430022a4546f784fc2669a3e11f2c 2023-03-26 11:56:05,756 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/8f50bc98398d45f7be237f2379ab2437 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8f50bc98398d45f7be237f2379ab2437 2023-03-26 11:56:05,763 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8f50bc98398d45f7be237f2379ab2437, entries=150, sequenceid=123, filesize=25.1 K 2023-03-26 11:56:05,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/a7e6bb3552614e7284b84c095117909a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a7e6bb3552614e7284b84c095117909a 2023-03-26 11:56:05,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a7e6bb3552614e7284b84c095117909a, entries=150, sequenceid=123, filesize=11.6 K 2023-03-26 11:56:05,767 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/b5b430022a4546f784fc2669a3e11f2c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/b5b430022a4546f784fc2669a3e11f2c 2023-03-26 11:56:05,770 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/b5b430022a4546f784fc2669a3e11f2c, entries=150, sequenceid=123, filesize=11.6 K 2023-03-26 11:56:05,771 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5890b64a473c744d0cc3b74eead11b50 in 59ms, sequenceid=123, compaction requested=true 2023-03-26 11:56:05,771 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:05,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:05,771 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:05,771 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:05,771 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:05,773 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35864 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:05,773 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:05,773 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,773 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a2dd2c5b36e94ac6b7b608bd6f6c42f2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/1965077d684d4082b064054d4dc2e125, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a7e6bb3552614e7284b84c095117909a] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=35.0 K 2023-03-26 11:56:05,773 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 77444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:05,773 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:05,773 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a2dd2c5b36e94ac6b7b608bd6f6c42f2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679831764155 2023-03-26 11:56:05,773 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,774 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a66104a2c84e4700bbc346ca5e2c17b5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96ff62b902f34b3dae9155e469aba053, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8f50bc98398d45f7be237f2379ab2437] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=75.6 K 2023-03-26 11:56:05,774 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 1965077d684d4082b064054d4dc2e125, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679831764276 2023-03-26 11:56:05,774 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a66104a2c84e4700bbc346ca5e2c17b5, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679831764155 2023-03-26 11:56:05,774 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a7e6bb3552614e7284b84c095117909a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679831765395 2023-03-26 11:56:05,775 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 96ff62b902f34b3dae9155e469aba053, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679831764276 2023-03-26 11:56:05,775 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8f50bc98398d45f7be237f2379ab2437, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679831765395 2023-03-26 11:56:05,781 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#523 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:05,798 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:05,799 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#524 average throughput is 1.23 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:05,801 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/bce4698fc8aa4c14bff4a6f3699d3bfe as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/bce4698fc8aa4c14bff4a6f3699d3bfe 2023-03-26 11:56:05,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:05,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:05,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:05,804 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:05,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:05,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:05,804 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:05,805 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,805 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,805 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into bce4698fc8aa4c14bff4a6f3699d3bfe(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:05,805 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,805 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831765771; duration=0sec 2023-03-26 11:56:05,806 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:05,806 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:05,806 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:05,806 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:05,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:05,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,807 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:05,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,807 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47682 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:05,807 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:05,807 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:05,807 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,807 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/4c973dfc185d4d2e8ecf0ae7e6222893, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/919bf69243124aa7a7bb80d426b355c4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a896df2ade074958b64fac5762d6dbc2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/b5b430022a4546f784fc2669a3e11f2c] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=46.6 K 2023-03-26 11:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:05,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:05,808 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 4c973dfc185d4d2e8ecf0ae7e6222893, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=69, earliestPutTs=1679831763530 2023-03-26 11:56:05,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:05,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:05,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:05,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:05,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:05,808 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 919bf69243124aa7a7bb80d426b355c4, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=85, earliestPutTs=1679831764155 2023-03-26 11:56:05,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:05,808 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:05,809 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:56:05,809 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a896df2ade074958b64fac5762d6dbc2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=105, earliestPutTs=1679831764276 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:05,809 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b5b430022a4546f784fc2669a3e11f2c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679831765395 2023-03-26 11:56:05,809 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:05,816 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#526 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:05,827 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a073a524f4a546c1948c08f8349eec5c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a073a524f4a546c1948c08f8349eec5c 2023-03-26 11:56:05,832 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/160784b63eb94d168df85140b7bbdd3f 2023-03-26 11:56:05,835 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/51731882efee419488dbaee555edac0a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/51731882efee419488dbaee555edac0a 2023-03-26 11:56:05,840 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 51731882efee419488dbaee555edac0a(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:05,840 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:05,840 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=12, startTime=1679831765771; duration=0sec 2023-03-26 11:56:05,840 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:05,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:05,840 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:05,853 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831825850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831825850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831825851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831825853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,856 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831825854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831825955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831825955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,956 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831825955, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,957 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831825956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:05,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:05,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831825956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831826158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831826158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831826158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,160 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831826159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,162 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831826160, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,225 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/4aa09191b44d4d48ab95d7842f476173 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4aa09191b44d4d48ab95d7842f476173 2023-03-26 11:56:06,230 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 4aa09191b44d4d48ab95d7842f476173(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:06,230 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:06,230 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831765771; duration=0sec 2023-03-26 11:56:06,230 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:06,247 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/de3e67ed33084a70baf9e1b92e6632c7 2023-03-26 11:56:06,262 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/af63d69742b34ffb83050a150f73da4c 2023-03-26 11:56:06,266 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/160784b63eb94d168df85140b7bbdd3f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/160784b63eb94d168df85140b7bbdd3f 2023-03-26 11:56:06,270 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/160784b63eb94d168df85140b7bbdd3f, entries=150, sequenceid=140, filesize=25.3 K 2023-03-26 11:56:06,270 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/de3e67ed33084a70baf9e1b92e6632c7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de3e67ed33084a70baf9e1b92e6632c7 2023-03-26 11:56:06,274 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de3e67ed33084a70baf9e1b92e6632c7, entries=150, sequenceid=140, filesize=11.8 K 2023-03-26 11:56:06,274 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/af63d69742b34ffb83050a150f73da4c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/af63d69742b34ffb83050a150f73da4c 2023-03-26 11:56:06,278 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/af63d69742b34ffb83050a150f73da4c, entries=150, sequenceid=140, filesize=11.8 K 2023-03-26 11:56:06,281 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5890b64a473c744d0cc3b74eead11b50 in 472ms, sequenceid=140, compaction requested=false 2023-03-26 11:56:06,281 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:06,281 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:06,281 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:06,281 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:06,281 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,281 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:06,281 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:06,285 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:06,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:06,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:06,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:06,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:06,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e4ec225[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:06,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:06,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,289 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:06,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:06,289 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:06,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,290 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:06,291 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:06,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:06,292 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:06,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:06,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:06,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:06,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:06,295 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:06,295 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:06,295 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:06,295 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:06,295 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:06,297 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,297 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:06,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,297 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:06,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:06,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:06,297 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:06,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:06,299 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:06,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:06,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:06,301 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:06,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,302 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:06,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:06,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:06,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:06,302 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:06,303 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:06,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:06,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:06,303 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:06,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:06,403 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:06,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:06,405 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:06,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:06,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:06,409 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:06,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:06,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:06,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:06,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,410 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:06,411 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:06,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:06,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:06,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:06,412 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:06,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:06,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:06,412 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:06,412 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:06,413 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:06,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:06,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:06,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:06,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:06,413 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:06,413 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:06,413 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:06,441 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303264ad61d2ffe344a25bc57f241d3a7b611 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264ad61d2ffe344a25bc57f241d3a7b611 2023-03-26 11:56:06,446 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=158, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/0e18c695bf044b4495af2374a2a17005 2023-03-26 11:56:06,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:06,464 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:06,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831826471, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,474 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831826472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831826472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831826473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,477 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831826474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831826575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831826575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831826577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831826577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,580 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831826577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831826778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,780 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831826778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831826780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831826780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,784 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:06,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831826781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:06,855 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/c28d561e7a2e4023b58472ead25dd1b4 2023-03-26 11:56:07,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831827081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831827081, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,084 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831827082, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,087 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831827084, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,088 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831827086, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,266 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=158 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/d4da6d34945348b78e397990746881e6 2023-03-26 11:56:07,270 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/0e18c695bf044b4495af2374a2a17005 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0e18c695bf044b4495af2374a2a17005 2023-03-26 11:56:07,273 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0e18c695bf044b4495af2374a2a17005, entries=150, sequenceid=158, filesize=25.3 K 2023-03-26 11:56:07,274 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/c28d561e7a2e4023b58472ead25dd1b4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c28d561e7a2e4023b58472ead25dd1b4 2023-03-26 11:56:07,277 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c28d561e7a2e4023b58472ead25dd1b4, entries=150, sequenceid=158, filesize=11.8 K 2023-03-26 11:56:07,278 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/d4da6d34945348b78e397990746881e6 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d4da6d34945348b78e397990746881e6 2023-03-26 11:56:07,282 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d4da6d34945348b78e397990746881e6, entries=150, sequenceid=158, filesize=11.8 K 2023-03-26 11:56:07,282 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5890b64a473c744d0cc3b74eead11b50 in 869ms, sequenceid=158, compaction requested=true 2023-03-26 11:56:07,282 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:07,283 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:07,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:07,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:07,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:07,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:07,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5700614d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:07,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:07,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:07,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:07,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:07,288 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:07,290 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:07,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:07,290 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:07,290 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:07,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:07,292 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:07,291 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:07,292 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:07,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:07,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:07,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:07,300 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:07,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:07,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:07,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:07,303 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:07,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:07,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:07,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:07,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:07,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,307 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:07,307 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:07,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:07,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:07,307 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,308 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:07,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:07,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:07,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:07,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:07,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:07,308 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:07,309 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:07,309 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:07,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:07,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:07,409 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:07,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:07,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:07,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:07,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:07,412 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:07,412 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:07,412 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:07,412 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:07,413 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,413 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:07,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:07,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:07,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:07,415 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:07,415 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,415 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:07,415 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:07,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:07,415 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:07,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:07,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:07,416 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:07,416 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:07,416 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:07,416 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:56:07,416 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:07,417 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:07,417 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:07,417 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:07,417 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:07,417 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:07,433 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326ddb172ddf6fd466c8c2e9cca9e978210 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ddb172ddf6fd466c8c2e9cca9e978210 2023-03-26 11:56:07,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:07,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:07,601 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831827597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,601 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,601 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831827599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831827600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,603 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831827600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,604 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831827601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831827702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,704 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831827702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831827704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,707 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831827704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831827705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,842 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=174, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/aceac7e737964727b66a59f9d3a348f9 2023-03-26 11:56:07,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831827905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,908 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831827905, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831827908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831827908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:07,912 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:07,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831827910, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831828210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831828210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,212 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831828211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,213 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831828211, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,216 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831828215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,252 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/7ad7a215de564db9809c62e433b7b398 2023-03-26 11:56:08,261 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=174 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/49139d2f6c984b219f922d19af20e93d 2023-03-26 11:56:08,265 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/aceac7e737964727b66a59f9d3a348f9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/aceac7e737964727b66a59f9d3a348f9 2023-03-26 11:56:08,268 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/aceac7e737964727b66a59f9d3a348f9, entries=150, sequenceid=174, filesize=25.3 K 2023-03-26 11:56:08,269 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/7ad7a215de564db9809c62e433b7b398 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7ad7a215de564db9809c62e433b7b398 2023-03-26 11:56:08,272 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7ad7a215de564db9809c62e433b7b398, entries=150, sequenceid=174, filesize=11.8 K 2023-03-26 11:56:08,272 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/49139d2f6c984b219f922d19af20e93d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/49139d2f6c984b219f922d19af20e93d 2023-03-26 11:56:08,275 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/49139d2f6c984b219f922d19af20e93d, entries=150, sequenceid=174, filesize=11.8 K 2023-03-26 11:56:08,276 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 5890b64a473c744d0cc3b74eead11b50 in 860ms, sequenceid=174, compaction requested=true 2023-03-26 11:56:08,276 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:08,276 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:08,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:08,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:08,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:08,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:08,279 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,279 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:08,279 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,279 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:08,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:08,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:08,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7263599a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:08,281 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:08,281 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,283 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:08,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:08,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:08,284 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:08,284 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,285 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:08,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:08,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:08,287 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:08,287 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:08,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:08,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:08,287 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:08,287 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:08,290 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,290 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:08,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,290 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:08,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:08,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,291 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:08,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:08,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:08,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:08,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:08,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:08,300 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:08,300 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:08,301 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:08,301 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:08,301 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:08,301 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:08,301 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:08,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:08,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:08,401 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:08,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:08,403 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:08,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:08,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:08,405 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:08,408 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:08,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:08,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:08,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,409 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:08,409 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:08,410 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:08,410 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:08,410 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:08,410 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:08,410 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:08,410 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-03-26 11:56:08,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:08,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:08,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:08,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:08,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:08,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:08,426 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032606aa51b714af4d91ae89ce2aeb1f1867 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032606aa51b714af4d91ae89ce2aeb1f1867 2023-03-26 11:56:08,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:08,716 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:08,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831828726, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,743 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831828727, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831828741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831828743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831828743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,831 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=29.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/7e4ffe09120d450bad852bc61e9a5444 2023-03-26 11:56:08,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831828843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831828844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831828848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831828848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:08,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:08,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831828848, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831829048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831829048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831829052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831829052, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831829053, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,241 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/da69bc1a60434f3996679aea86b9611c 2023-03-26 11:56:09,252 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/bd372eaa11b64a259c86e40c7548e269 2023-03-26 11:56:09,256 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/7e4ffe09120d450bad852bc61e9a5444 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7e4ffe09120d450bad852bc61e9a5444 2023-03-26 11:56:09,260 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7e4ffe09120d450bad852bc61e9a5444, entries=150, sequenceid=190, filesize=25.3 K 2023-03-26 11:56:09,261 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/da69bc1a60434f3996679aea86b9611c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/da69bc1a60434f3996679aea86b9611c 2023-03-26 11:56:09,264 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/da69bc1a60434f3996679aea86b9611c, entries=150, sequenceid=190, filesize=11.8 K 2023-03-26 11:56:09,265 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/bd372eaa11b64a259c86e40c7548e269 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd372eaa11b64a259c86e40c7548e269 2023-03-26 11:56:09,268 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd372eaa11b64a259c86e40c7548e269, entries=150, sequenceid=190, filesize=11.8 K 2023-03-26 11:56:09,269 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 5890b64a473c744d0cc3b74eead11b50 in 859ms, sequenceid=190, compaction requested=true 2023-03-26 11:56:09,269 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:09,269 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:09,269 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:09,271 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,271 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:09,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ee1741b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:09,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:09,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,275 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:09,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:09,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:09,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:09,277 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,277 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:09,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,277 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,277 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:09,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:09,279 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:09,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:09,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:09,279 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:09,279 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:09,281 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,281 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,281 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,282 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,283 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,286 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,286 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,286 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,286 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,286 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:09,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:09,286 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,287 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:09,287 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:09,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:09,353 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 11:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:09,355 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:09,368 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326f6d286b9f3584942a8092ab82fd8d18d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f6d286b9f3584942a8092ab82fd8d18d 2023-03-26 11:56:09,368 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831829363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831829364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,369 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831829364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831829366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831829366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,387 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:09,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:09,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:09,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:09,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:09,390 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:09,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:09,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:09,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,391 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,392 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,392 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,393 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,393 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,393 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:09,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:09,394 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:09,394 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:09,394 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,394 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:09,394 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:09,394 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:09,394 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:09,395 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:09,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,398 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:09,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:09,400 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16a6d008[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:09,400 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:09,400 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,402 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:09,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:09,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:09,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:09,407 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:09,407 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:09,407 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:09,407 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:09,407 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:09,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,409 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,410 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:09,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,414 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:09,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,414 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,415 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:09,415 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:09,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,473 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831829469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831829470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,474 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831829470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831829472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,475 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831829472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,515 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:09,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:09,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:09,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:09,518 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:09,518 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,518 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:09,518 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:09,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,521 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,521 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,521 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,521 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:09,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,521 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:09,522 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:09,522 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:09,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:09,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:09,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:09,522 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:09,522 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,522 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:09,522 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:09,523 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:09,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:09,525 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,525 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:09,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4278c198[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:09,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:09,527 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:09,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:09,529 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:09,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,529 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:09,531 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,531 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,531 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,531 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:09,533 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:09,533 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:09,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:09,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:09,533 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:09,533 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:09,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,535 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,538 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,540 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,540 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,540 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,540 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,541 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:09,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:09,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,541 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:09,541 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:09,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,641 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:09,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:09,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:09,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:09,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:09,645 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:09,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:09,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:09,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,646 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,647 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:09,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,648 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:09,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:09,648 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,648 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:09,648 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:09,648 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:09,648 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:09,649 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:09,655 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,655 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:09,656 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2abfefc4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:09,656 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:09,657 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,658 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,658 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:09,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:09,658 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:09,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,658 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:09,659 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,659 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:09,661 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:09,661 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:09,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:09,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:09,662 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:09,662 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:09,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,665 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,665 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,666 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,669 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:09,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,669 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831829675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831829675, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,677 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831829676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,677 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831829676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,678 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:09,678 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:09,678 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:09,678 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:09,681 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831829678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,778 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:09,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:09,779 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=209, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec 2023-03-26 11:56:09,780 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:09,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:09,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:09,782 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:09,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:09,782 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:09,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,783 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,784 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,784 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:09,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:09,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:09,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,786 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:09,787 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:09,787 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:09,789 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,789 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:09,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:09,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:09,791 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f7b0724[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:09,791 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,792 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:09,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:09,793 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:09,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,793 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:09,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:09,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,794 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,794 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:09,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:09,797 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:09,797 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:09,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:09,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:09,797 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:09,797 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:09,799 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,799 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,799 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,801 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,802 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,805 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:09,805 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,805 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,806 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:09,805 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,806 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:09,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,806 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,906 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:09,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:09,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:09,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:09,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:09,910 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:09,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:09,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:09,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,913 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,913 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,914 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:09,914 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:09,914 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:09,914 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:09,914 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:09,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:09,915 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:09,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:09,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:09,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:09,915 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:09,916 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,917 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:09,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ffda453[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:09,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:09,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,920 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,920 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:09,920 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:09,920 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:09,920 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,920 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:09,922 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:09,922 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,922 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:09,923 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:09,925 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:09,925 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:09,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:09,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:09,925 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:09,925 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:09,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:09,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,927 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:09,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,928 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:09,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,929 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:09,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:09,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:09,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:09,931 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:09,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:09,931 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:09,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:09,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:09,932 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,932 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:09,932 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:09,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:09,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:09,932 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:09,932 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:09,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831829978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831829978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,981 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831829980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,982 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831829980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:09,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:09,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831829983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,032 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:10,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:10,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:10,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:10,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:10,036 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:10,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:10,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:10,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,036 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,037 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,038 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,039 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:10,039 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:10,039 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:10,039 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,039 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:10,039 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:10,039 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:10,039 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,039 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:10,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:10,040 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:10,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:10,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:10,040 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:10,041 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,042 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:10,042 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:10,044 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b0dd035[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:10,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:10,044 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,049 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:10,049 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:10,049 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,049 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:10,051 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:10,051 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,051 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,051 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:10,054 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:10,054 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:10,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:10,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:10,054 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:10,054 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:10,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,056 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,056 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,058 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,062 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:10,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,062 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,062 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:10,062 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:10,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,162 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:10,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:10,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:10,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:10,166 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:10,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:10,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:10,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:10,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:10,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:10,170 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:10,170 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:10,170 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:10,170 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,170 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:10,171 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:10,171 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:10,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:10,172 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,173 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:10,174 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e19cf5f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:10,174 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:10,174 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,175 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:10,176 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:10,176 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,176 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:10,178 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,178 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:10,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:10,181 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:10,181 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:10,181 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:10,181 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:10,181 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:10,184 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,184 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,184 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,186 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,189 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:10,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,189 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:10,189 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:10,190 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:10,194 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/fe8a1bc2798d42a8bf64010aaf759439 2023-03-26 11:56:10,214 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=209 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c8ebffafb3ad45908b67c1b288ec8f45 2023-03-26 11:56:10,220 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec 2023-03-26 11:56:10,225 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec, entries=200, sequenceid=209, filesize=32.1 K 2023-03-26 11:56:10,226 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/fe8a1bc2798d42a8bf64010aaf759439 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/fe8a1bc2798d42a8bf64010aaf759439 2023-03-26 11:56:10,229 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/fe8a1bc2798d42a8bf64010aaf759439, entries=150, sequenceid=209, filesize=11.8 K 2023-03-26 11:56:10,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c8ebffafb3ad45908b67c1b288ec8f45 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c8ebffafb3ad45908b67c1b288ec8f45 2023-03-26 11:56:10,233 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c8ebffafb3ad45908b67c1b288ec8f45, entries=150, sequenceid=209, filesize=11.8 K 2023-03-26 11:56:10,234 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 5890b64a473c744d0cc3b74eead11b50 in 881ms, sequenceid=209, compaction requested=true 2023-03-26 11:56:10,234 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:10,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:10,234 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:56:10,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:10,234 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:56:10,234 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:10,236 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 162524 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:56:10,237 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:10,237 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,237 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4aa09191b44d4d48ab95d7842f476173, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/160784b63eb94d168df85140b7bbdd3f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0e18c695bf044b4495af2374a2a17005, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/aceac7e737964727b66a59f9d3a348f9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7e4ffe09120d450bad852bc61e9a5444, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=158.7 K 2023-03-26 11:56:10,237 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72374 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:56:10,237 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:10,237 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,237 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/bce4698fc8aa4c14bff4a6f3699d3bfe, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de3e67ed33084a70baf9e1b92e6632c7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c28d561e7a2e4023b58472ead25dd1b4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7ad7a215de564db9809c62e433b7b398, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/da69bc1a60434f3996679aea86b9611c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/fe8a1bc2798d42a8bf64010aaf759439] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=70.7 K 2023-03-26 11:56:10,238 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 4aa09191b44d4d48ab95d7842f476173, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679831765395 2023-03-26 11:56:10,238 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bce4698fc8aa4c14bff4a6f3699d3bfe, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679831765395 2023-03-26 11:56:10,238 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 160784b63eb94d168df85140b7bbdd3f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1679831765730 2023-03-26 11:56:10,238 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting de3e67ed33084a70baf9e1b92e6632c7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1679831765730 2023-03-26 11:56:10,238 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0e18c695bf044b4495af2374a2a17005, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679831765850 2023-03-26 11:56:10,239 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c28d561e7a2e4023b58472ead25dd1b4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679831765850 2023-03-26 11:56:10,239 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting aceac7e737964727b66a59f9d3a348f9, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679831766469 2023-03-26 11:56:10,240 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 7ad7a215de564db9809c62e433b7b398, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679831766469 2023-03-26 11:56:10,240 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 7e4ffe09120d450bad852bc61e9a5444, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679831767599 2023-03-26 11:56:10,240 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting da69bc1a60434f3996679aea86b9611c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679831767599 2023-03-26 11:56:10,240 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679831768728 2023-03-26 11:56:10,240 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fe8a1bc2798d42a8bf64010aaf759439, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679831768728 2023-03-26 11:56:10,254 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#542 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:10,259 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#541 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:10,265 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/9cd5c9aff0164f679370f990b3431f32 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9cd5c9aff0164f679370f990b3431f32 2023-03-26 11:56:10,270 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 9cd5c9aff0164f679370f990b3431f32(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:10,271 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:10,271 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=10, startTime=1679831770234; duration=0sec 2023-03-26 11:56:10,271 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:10,271 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-03-26 11:56:10,272 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 72340 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-03-26 11:56:10,272 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:10,272 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,273 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/51731882efee419488dbaee555edac0a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/af63d69742b34ffb83050a150f73da4c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d4da6d34945348b78e397990746881e6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/49139d2f6c984b219f922d19af20e93d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd372eaa11b64a259c86e40c7548e269, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c8ebffafb3ad45908b67c1b288ec8f45] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=70.6 K 2023-03-26 11:56:10,273 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 51731882efee419488dbaee555edac0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=123, earliestPutTs=1679831765395 2023-03-26 11:56:10,273 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting af63d69742b34ffb83050a150f73da4c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1679831765730 2023-03-26 11:56:10,273 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d4da6d34945348b78e397990746881e6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=158, earliestPutTs=1679831765850 2023-03-26 11:56:10,274 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 49139d2f6c984b219f922d19af20e93d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=174, earliestPutTs=1679831766469 2023-03-26 11:56:10,274 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bd372eaa11b64a259c86e40c7548e269, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1679831767599 2023-03-26 11:56:10,274 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c8ebffafb3ad45908b67c1b288ec8f45, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679831768728 2023-03-26 11:56:10,286 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#543 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:10,289 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/04f0aa89a298451db9f3051b5db51959 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/04f0aa89a298451db9f3051b5db51959 2023-03-26 11:56:10,290 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:10,295 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 04f0aa89a298451db9f3051b5db51959(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:10,295 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:10,295 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=10, startTime=1679831770234; duration=0sec 2023-03-26 11:56:10,295 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:10,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:10,299 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:10,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:10,300 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:10,300 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:10,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:10,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:10,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,302 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,302 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,304 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:10,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,304 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:10,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:10,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:10,307 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:10,307 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,307 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:10,307 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:10,308 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:10,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:10,485 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831830494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,498 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831830495, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,499 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831830496, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,499 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831830497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,500 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,501 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831830498, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831830599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,602 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831830599, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831830600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,602 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831830600, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831830601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,701 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/dd67ee546a6b419a8fc6a5f7e0505e0a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/dd67ee546a6b419a8fc6a5f7e0505e0a 2023-03-26 11:56:10,704 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into dd67ee546a6b419a8fc6a5f7e0505e0a(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:10,704 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:10,704 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=10, startTime=1679831770234; duration=0sec 2023-03-26 11:56:10,704 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:10,725 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a9af9e2540fe420e9c9eeae5596fc190 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a9af9e2540fe420e9c9eeae5596fc190 2023-03-26 11:56:10,730 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=224, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/6ab445453111461db271a11c54fb5078 2023-03-26 11:56:10,739 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/13a95957b2ac4034bdfe50304316f466 2023-03-26 11:56:10,749 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=224 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/68dac10c824b4737b6ea73eaa7d2f567 2023-03-26 11:56:10,752 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/6ab445453111461db271a11c54fb5078 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/6ab445453111461db271a11c54fb5078 2023-03-26 11:56:10,756 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/6ab445453111461db271a11c54fb5078, entries=150, sequenceid=224, filesize=25.3 K 2023-03-26 11:56:10,756 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/13a95957b2ac4034bdfe50304316f466 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13a95957b2ac4034bdfe50304316f466 2023-03-26 11:56:10,759 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13a95957b2ac4034bdfe50304316f466, entries=150, sequenceid=224, filesize=11.8 K 2023-03-26 11:56:10,760 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/68dac10c824b4737b6ea73eaa7d2f567 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68dac10c824b4737b6ea73eaa7d2f567 2023-03-26 11:56:10,763 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68dac10c824b4737b6ea73eaa7d2f567, entries=150, sequenceid=224, filesize=11.8 K 2023-03-26 11:56:10,764 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 5890b64a473c744d0cc3b74eead11b50 in 457ms, sequenceid=224, compaction requested=false 2023-03-26 11:56:10,764 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:10,764 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:10,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:10,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:10,764 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:10,767 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,767 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,767 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,767 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:10,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292ca19f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:10,770 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:10,770 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,771 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:10,771 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:10,771 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,772 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:10,774 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,774 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:10,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:10,777 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:10,777 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:10,777 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:10,777 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:10,777 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:10,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,779 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,780 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,781 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,785 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:10,785 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:10,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:10,786 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:10,786 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:10,785 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,786 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:10,805 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-03-26 11:56:10,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:10,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:10,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:10,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:10,806 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:10,806 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:10,815 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831830811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,816 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831830812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831830814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831830815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831830816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,819 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263b58ae5fa02b46fbb42e7b68c1e702fc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263b58ae5fa02b46fbb42e7b68c1e702fc 2023-03-26 11:56:10,823 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/3b640976a5ef47528c43f42b23f398cb 2023-03-26 11:56:10,831 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/6877928b262649c0bc8a750c4cf928f1 2023-03-26 11:56:10,886 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:10,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:10,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:10,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:10,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:10,889 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:10,889 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,889 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:10,889 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:10,889 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,890 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,891 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,891 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,891 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:10,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:10,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:10,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:10,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:10,892 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:10,892 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:10,892 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:10,893 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,893 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:10,893 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:10,893 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:10,893 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:10,895 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,895 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,896 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:10,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1147d5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:10,897 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:10,897 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,899 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,899 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:10,899 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:10,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,899 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:10,901 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:10,901 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,901 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,901 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:10,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:10,904 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:10,904 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:10,905 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:10,905 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:10,905 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:10,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:10,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:10,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,909 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:10,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,910 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:10,912 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:10,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:10,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:10,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:10,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:10,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:10,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:10,914 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:10,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:10,914 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:10,915 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:10,915 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:10,915 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:10,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:10,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:10,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831830917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,918 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831830917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831830918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831830918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:10,921 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:10,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831830919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,015 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,018 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,020 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,020 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,021 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:11,022 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,022 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,023 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,025 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,025 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,026 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,026 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,027 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e78aafa[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,027 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,028 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,028 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,029 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,030 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,035 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,035 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,035 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,035 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,037 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,037 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,039 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,042 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,043 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,043 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,043 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,044 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,044 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831831120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35794 deadline: 1679831831120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,122 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35824 deadline: 1679831831122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831831123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831831123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,144 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,147 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,147 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,149 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,149 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,150 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,150 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,150 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:11,151 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,153 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,153 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,153 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,153 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e8bfb5d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,154 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,154 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,156 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,156 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,159 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,159 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,159 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,159 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,161 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,161 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,161 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,161 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,162 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,164 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,164 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,164 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,166 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,168 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,168 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,169 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:11,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,169 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,169 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,241 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/d611bb0d41144fa4a3b8f507a2945082 2023-03-26 11:56:11,244 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/3b640976a5ef47528c43f42b23f398cb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b640976a5ef47528c43f42b23f398cb 2023-03-26 11:56:11,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b640976a5ef47528c43f42b23f398cb, entries=200, sequenceid=245, filesize=32.1 K 2023-03-26 11:56:11,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/6877928b262649c0bc8a750c4cf928f1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/6877928b262649c0bc8a750c4cf928f1 2023-03-26 11:56:11,251 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/6877928b262649c0bc8a750c4cf928f1, entries=150, sequenceid=245, filesize=11.8 K 2023-03-26 11:56:11,252 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/d611bb0d41144fa4a3b8f507a2945082 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d611bb0d41144fa4a3b8f507a2945082 2023-03-26 11:56:11,255 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d611bb0d41144fa4a3b8f507a2945082, entries=150, sequenceid=245, filesize=11.8 K 2023-03-26 11:56:11,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 5890b64a473c744d0cc3b74eead11b50 in 450ms, sequenceid=245, compaction requested=true 2023-03-26 11:56:11,255 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:11,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:11,255 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:11,255 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:11,255 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:11,256 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:11,257 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85190 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:11,257 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36620 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:11,257 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:11,257 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:11,257 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,257 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,257 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/04f0aa89a298451db9f3051b5db51959, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/6ab445453111461db271a11c54fb5078, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b640976a5ef47528c43f42b23f398cb] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=83.2 K 2023-03-26 11:56:11,257 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9cd5c9aff0164f679370f990b3431f32, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13a95957b2ac4034bdfe50304316f466, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/6877928b262649c0bc8a750c4cf928f1] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=35.8 K 2023-03-26 11:56:11,257 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 04f0aa89a298451db9f3051b5db51959, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679831768728 2023-03-26 11:56:11,257 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9cd5c9aff0164f679370f990b3431f32, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679831768728 2023-03-26 11:56:11,258 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 6ab445453111461db271a11c54fb5078, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679831769357 2023-03-26 11:56:11,258 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 13a95957b2ac4034bdfe50304316f466, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679831769357 2023-03-26 11:56:11,258 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3b640976a5ef47528c43f42b23f398cb, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679831770495 2023-03-26 11:56:11,258 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6877928b262649c0bc8a750c4cf928f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679831770496 2023-03-26 11:56:11,269 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,273 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,273 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#551 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:11,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,274 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,275 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,275 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,277 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#550 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:11,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,278 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,278 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,278 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,278 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,278 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,278 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,278 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,279 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-03-26 11:56:11,279 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:11,279 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:11,279 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:11,279 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:11,279 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:11,279 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:11,317 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267b5dabb2feda456297142e9117f7e1ff to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267b5dabb2feda456297142e9117f7e1ff 2023-03-26 11:56:11,322 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=258, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/b72fa7503e0049f6a58e77fd7899f59d 2023-03-26 11:56:11,338 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/c3ea72e256284347809a46624b427fdd 2023-03-26 11:56:11,350 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=258 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/808d5d74ae614ec2a54c4f3766338ec7 2023-03-26 11:56:11,354 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/b72fa7503e0049f6a58e77fd7899f59d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b72fa7503e0049f6a58e77fd7899f59d 2023-03-26 11:56:11,357 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b72fa7503e0049f6a58e77fd7899f59d, entries=150, sequenceid=258, filesize=25.3 K 2023-03-26 11:56:11,358 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/c3ea72e256284347809a46624b427fdd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3ea72e256284347809a46624b427fdd 2023-03-26 11:56:11,361 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3ea72e256284347809a46624b427fdd, entries=150, sequenceid=258, filesize=11.8 K 2023-03-26 11:56:11,361 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/808d5d74ae614ec2a54c4f3766338ec7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/808d5d74ae614ec2a54c4f3766338ec7 2023-03-26 11:56:11,364 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/808d5d74ae614ec2a54c4f3766338ec7, entries=150, sequenceid=258, filesize=11.8 K 2023-03-26 11:56:11,365 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for 5890b64a473c744d0cc3b74eead11b50 in 86ms, sequenceid=258, compaction requested=true 2023-03-26 11:56:11,365 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:11,365 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,365 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,369 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,372 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,372 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72e7954a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,374 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,375 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,375 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,377 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,380 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,380 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,380 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,381 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,390 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:11,390 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,391 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,391 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:11,449 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:56:11,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:11,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:11,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:11,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:11,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:11,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:11,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831831465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831831465, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831771459, max timestamp=1679831771469Num uniques 859; 2023-03-26 11:56:11,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:11,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831771459, max timestamp=1679831771469Num uniques 859; 2023-03-26 11:56:11,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831771459, max timestamp=1679831771469Num uniques 859; 2023-03-26 11:56:11,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:11,471 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:11,491 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,492 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,494 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,494 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,495 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,497 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,497 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,497 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:11,497 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,497 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,499 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33894b2c[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,500 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,503 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,503 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,503 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,503 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,505 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,505 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,507 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,507 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,507 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,507 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,508 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,509 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,510 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:11,512 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,512 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,513 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,512 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,513 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831831519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,525 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35762 deadline: 1679831831520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831831520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,574 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831831571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831831572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,613 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,616 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,619 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,619 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,620 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,620 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,620 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:11,621 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,621 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,622 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,622 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,622 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831831621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,623 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,623 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@584d08dd[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,626 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,626 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,626 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,626 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,626 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,627 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,627 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,627 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,628 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831831626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831831627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,633 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,633 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,633 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,633 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,633 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,633 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,635 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,636 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,637 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,640 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,640 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,640 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,640 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,640 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,708 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/ccd1260f509744d8a90b60d2545d88bf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/ccd1260f509744d8a90b60d2545d88bf 2023-03-26 11:56:11,712 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into ccd1260f509744d8a90b60d2545d88bf(size=12.4 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:11,712 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:11,712 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831771255; duration=0sec 2023-03-26 11:56:11,712 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:11,712 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:11,715 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48672 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:11,715 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:11,715 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,715 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/dd67ee546a6b419a8fc6a5f7e0505e0a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68dac10c824b4737b6ea73eaa7d2f567, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d611bb0d41144fa4a3b8f507a2945082, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/808d5d74ae614ec2a54c4f3766338ec7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=47.5 K 2023-03-26 11:56:11,715 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dd67ee546a6b419a8fc6a5f7e0505e0a, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=209, earliestPutTs=1679831768728 2023-03-26 11:56:11,716 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 68dac10c824b4737b6ea73eaa7d2f567, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=224, earliestPutTs=1679831769357 2023-03-26 11:56:11,716 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d611bb0d41144fa4a3b8f507a2945082, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679831770496 2023-03-26 11:56:11,716 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 808d5d74ae614ec2a54c4f3766338ec7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679831770811 2023-03-26 11:56:11,722 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#556 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:11,740 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,743 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,744 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,745 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,745 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,745 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,745 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,745 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,746 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,746 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,746 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,746 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,746 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,746 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:11,746 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,746 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,748 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,748 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fd419a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,751 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,751 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,751 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,753 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,755 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,755 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,755 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,755 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,755 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,755 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,757 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,757 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,757 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,758 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:11,764 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,764 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,776 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35780 deadline: 1679831831775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831831776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,827 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:35802 deadline: 1679831831824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831831829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,832 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:11,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831831830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,864 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:11,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:11,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:11,868 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:11,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:11,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:11,868 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326e0dcb323ddc14e4caec6b6959eb43045 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e0dcb323ddc14e4caec6b6959eb43045 2023-03-26 11:56:11,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,870 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:11,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:11,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:11,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:11,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:11,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:11,872 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,872 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:11,872 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:11,872 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:11,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:11,873 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=269, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/cf965ca273d44e0d902a914ce99c2be8 2023-03-26 11:56:11,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:11,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@777ede2a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:11,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:11,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,878 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:11,878 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:11,878 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,878 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:11,880 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:11,881 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,881 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:11,881 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,883 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/2c91919bbf124473a0d700038222f6a0 2023-03-26 11:56:11,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:11,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:11,884 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:11,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:11,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:11,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:11,884 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:11,886 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,886 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:11,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,886 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:11,887 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,888 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:11,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:11,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,893 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=269 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/a65101d67253470fadecf451c75092fb 2023-03-26 11:56:11,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:11,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:11,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:11,896 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:11,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:11,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:11,896 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:11,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:11,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:11,897 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:11,897 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:11,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:11,897 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:11,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:11,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:11,898 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/cf965ca273d44e0d902a914ce99c2be8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/cf965ca273d44e0d902a914ce99c2be8 2023-03-26 11:56:11,901 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/cf965ca273d44e0d902a914ce99c2be8, entries=150, sequenceid=269, filesize=25.4 K 2023-03-26 11:56:11,902 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/2c91919bbf124473a0d700038222f6a0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2c91919bbf124473a0d700038222f6a0 2023-03-26 11:56:11,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2c91919bbf124473a0d700038222f6a0, entries=150, sequenceid=269, filesize=11.9 K 2023-03-26 11:56:11,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/a65101d67253470fadecf451c75092fb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a65101d67253470fadecf451c75092fb 2023-03-26 11:56:11,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a65101d67253470fadecf451c75092fb, entries=150, sequenceid=269, filesize=11.9 K 2023-03-26 11:56:11,909 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 5890b64a473c744d0cc3b74eead11b50 in 460ms, sequenceid=269, compaction requested=true 2023-03-26 11:56:11,909 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:11,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:11,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-03-26 11:56:11,909 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 11:56:11,997 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:11,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:11,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:12,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:12,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:12,000 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:12,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:12,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:12,000 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:12,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,002 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:12,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:12,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,002 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:12,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:12,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:12,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:12,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:12,004 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:12,004 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:12,024 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a0110ef7141e47d28ff1a89f3922356f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a0110ef7141e47d28ff1a89f3922356f 2023-03-26 11:56:12,033 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=307, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/0cbc503d1a6c4c38b42327c2313f857e 2023-03-26 11:56:12,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:12,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:12,109 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831832104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,115 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/8159cc6a8bd4436db877ddc3896b338a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8159cc6a8bd4436db877ddc3896b338a 2023-03-26 11:56:12,116 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831832112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,119 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 8159cc6a8bd4436db877ddc3896b338a(size=25.9 K), total size for store is 76.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:12,119 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:12,119 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831771255; duration=0sec 2023-03-26 11:56:12,119 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=3), splitQueue=0 2023-03-26 11:56:12,119 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 4 compacting, 1 eligible, 16 blocking 2023-03-26 11:56:12,119 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-03-26 11:56:12,120 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-03-26 11:56:12,120 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. because compaction request was cancelled 2023-03-26 11:56:12,120 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:12,120 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 78502 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:12,120 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:12,121 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:12,121 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8159cc6a8bd4436db877ddc3896b338a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b72fa7503e0049f6a58e77fd7899f59d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/cf965ca273d44e0d902a914ce99c2be8] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=76.7 K 2023-03-26 11:56:12,121 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8159cc6a8bd4436db877ddc3896b338a, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679831770496 2023-03-26 11:56:12,121 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b72fa7503e0049f6a58e77fd7899f59d, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679831770811 2023-03-26 11:56:12,121 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting cf965ca273d44e0d902a914ce99c2be8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679831771448 2023-03-26 11:56:12,129 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#561 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:12,130 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831832128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,131 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/68ff5dc54ef14f7bbf4060b67f1ac9a9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68ff5dc54ef14f7bbf4060b67f1ac9a9 2023-03-26 11:56:12,136 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 68ff5dc54ef14f7bbf4060b67f1ac9a9(size=12.4 K), total size for store is 24.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:12,136 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:12,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831832133, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,136 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=12, startTime=1679831771256; duration=0sec 2023-03-26 11:56:12,137 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:12,137 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:12,137 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36922 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:12,138 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:12,138 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:12,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,138 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/ccd1260f509744d8a90b60d2545d88bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3ea72e256284347809a46624b427fdd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2c91919bbf124473a0d700038222f6a0] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=36.1 K 2023-03-26 11:56:12,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831832134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,138 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting ccd1260f509744d8a90b60d2545d88bf, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1679831770496 2023-03-26 11:56:12,138 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c3ea72e256284347809a46624b427fdd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679831770811 2023-03-26 11:56:12,139 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 2c91919bbf124473a0d700038222f6a0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679831771448 2023-03-26 11:56:12,144 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#562 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:12,152 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/b05b4b46bdea42369b1c581f7dd07df8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b05b4b46bdea42369b1c581f7dd07df8 2023-03-26 11:56:12,155 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/12d9e1c0b50648fb9ff1a080050e1446 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/12d9e1c0b50648fb9ff1a080050e1446 2023-03-26 11:56:12,156 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into b05b4b46bdea42369b1c581f7dd07df8(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:12,156 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:12,156 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831771909; duration=0sec 2023-03-26 11:56:12,156 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:12,159 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 12d9e1c0b50648fb9ff1a080050e1446(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:12,159 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:12,159 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831771909; duration=0sec 2023-03-26 11:56:12,159 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:12,212 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831832210, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,219 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831832217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,415 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831832413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,422 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831832421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,446 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/7fe857a540e64a62abe4213442c9c0fb 2023-03-26 11:56:12,636 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831832634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,638 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831832637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,641 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831832640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831832716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,725 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:12,725 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831832723, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,855 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=307 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/d9b831d328e54cf784913467e6c6e215 2023-03-26 11:56:12,860 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/0cbc503d1a6c4c38b42327c2313f857e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0cbc503d1a6c4c38b42327c2313f857e 2023-03-26 11:56:12,863 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0cbc503d1a6c4c38b42327c2313f857e, entries=150, sequenceid=307, filesize=25.4 K 2023-03-26 11:56:12,864 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/7fe857a540e64a62abe4213442c9c0fb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7fe857a540e64a62abe4213442c9c0fb 2023-03-26 11:56:12,867 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7fe857a540e64a62abe4213442c9c0fb, entries=150, sequenceid=307, filesize=11.9 K 2023-03-26 11:56:12,868 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/d9b831d328e54cf784913467e6c6e215 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d9b831d328e54cf784913467e6c6e215 2023-03-26 11:56:12,871 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d9b831d328e54cf784913467e6c6e215, entries=150, sequenceid=307, filesize=11.9 K 2023-03-26 11:56:12,872 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 5890b64a473c744d0cc3b74eead11b50 in 868ms, sequenceid=307, compaction requested=true 2023-03-26 11:56:12,872 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:12,872 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:12,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:12,875 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,875 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:12,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:12,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@e572bc2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:12,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:12,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:12,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:12,879 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,879 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:12,880 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:12,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,880 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:12,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:12,881 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:12,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:12,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:12,883 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:12,883 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:12,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:12,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:12,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:12,883 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:12,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,885 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:12,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,886 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:12,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:12,888 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,891 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:12,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:12,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:12,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:12,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:12,891 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:12,891 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,891 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,892 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:12,892 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,991 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:12,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:12,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:12,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:12,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:12,995 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:12,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:12,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:12,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:12,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,997 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:12,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:12,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:12,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:12,997 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:12,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,998 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:12,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:12,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:12,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:12,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:12,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:12,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:12,999 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:13,000 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:13,000 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:13,033 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a65ed4bd1a4440d0ac35db33ebe74f86 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a65ed4bd1a4440d0ac35db33ebe74f86 2023-03-26 11:56:13,042 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=321, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/c113121e841e44e89885641e37b9f7a5 2023-03-26 11:56:13,059 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/dad75ada66c3412fb04ca39c00aefc83 2023-03-26 11:56:13,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:13,223 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:13,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831773236, max timestamp=1679831773251Num uniques 859; 2023-03-26 11:56:13,253 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:13,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831773236, max timestamp=1679831773251Num uniques 859; 2023-03-26 11:56:13,254 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:13,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831773236, max timestamp=1679831773251Num uniques 859; 2023-03-26 11:56:13,255 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:13,281 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831833278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831833280, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831833382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,385 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831833383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,468 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=321 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/7c5b61063652463692c0e12b93202a84 2023-03-26 11:56:13,472 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/c113121e841e44e89885641e37b9f7a5 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c113121e841e44e89885641e37b9f7a5 2023-03-26 11:56:13,475 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c113121e841e44e89885641e37b9f7a5, entries=150, sequenceid=321, filesize=25.4 K 2023-03-26 11:56:13,476 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/dad75ada66c3412fb04ca39c00aefc83 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/dad75ada66c3412fb04ca39c00aefc83 2023-03-26 11:56:13,479 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/dad75ada66c3412fb04ca39c00aefc83, entries=150, sequenceid=321, filesize=11.9 K 2023-03-26 11:56:13,480 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/7c5b61063652463692c0e12b93202a84 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/7c5b61063652463692c0e12b93202a84 2023-03-26 11:56:13,483 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/7c5b61063652463692c0e12b93202a84, entries=150, sequenceid=321, filesize=11.9 K 2023-03-26 11:56:13,483 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=131.94 KB/135110 for 5890b64a473c744d0cc3b74eead11b50 in 483ms, sequenceid=321, compaction requested=true 2023-03-26 11:56:13,483 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:13,483 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:13,484 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:13,485 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,485 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,486 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,486 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:13,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3e6cd135[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:13,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:13,487 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,488 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:13,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:13,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:13,490 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,490 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,490 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,490 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:13,493 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:13,493 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:13,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:13,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:13,493 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:13,493 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:13,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,494 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,498 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:13,498 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:13,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:13,499 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,499 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:13,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:13,589 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-03-26 11:56:13,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679831773274, max timestamp=1679831773589Num uniques 1068; 2023-03-26 11:56:13,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679831773274, max timestamp=1679831773589Num uniques 1068; 2023-03-26 11:56:13,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:13,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:13,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:13,592 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:13,592 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:13,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 11:56:13,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679831773274, max timestamp=1679831773589Num uniques 1018; 2023-03-26 11:56:13,592 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 11:56:13,593 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:13,593 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-03-26 11:56:13,599 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:13,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:13,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:13,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:13,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:13,602 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:13,602 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,602 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:13,602 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:13,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,604 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,604 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:13,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:13,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:13,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:13,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:13,606 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:13,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:13,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:13,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:13,611 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,611 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,612 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:13,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:13,613 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6dcea6f3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:13,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:13,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,615 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,615 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:13,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:13,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:13,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,615 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:13,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:13,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,617 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:13,617 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:13,620 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:13,620 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:13,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:13,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:13,620 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:13,620 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:13,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,622 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,622 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,623 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:13,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:13,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,628 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:13,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,628 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:13,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831773617, max timestamp=1679831773630Num uniques 859; 2023-03-26 11:56:13,631 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:13,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831773617, max timestamp=1679831773630Num uniques 859; 2023-03-26 11:56:13,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:13,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831773617, max timestamp=1679831773630Num uniques 859; 2023-03-26 11:56:13,632 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:13,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679831773645, max timestamp=1679831773651Num uniques 1068; 2023-03-26 11:56:13,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679831773645, max timestamp=1679831773651Num uniques 1068; 2023-03-26 11:56:13,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-26 11:56:13,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1679831773645, max timestamp=1679831773651Num uniques 1068; 2023-03-26 11:56:13,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-26 11:56:13,654 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-03-26 11:56:13,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679831773687, max timestamp=1679831773692Num uniques 1127; 2023-03-26 11:56:13,694 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-26 11:56:13,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679831773687, max timestamp=1679831773692Num uniques 1127; 2023-03-26 11:56:13,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1679831773687, max timestamp=1679831773692Num uniques 1127; 2023-03-26 11:56:13,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-26 11:56:13,695 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=10392, suffix heap size=145176,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=227 2023-03-26 11:56:13,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679831773703, max timestamp=1679831773711Num uniques 1086; 2023-03-26 11:56:13,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679831773703, max timestamp=1679831773711Num uniques 1086; 2023-03-26 11:56:13,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-26 11:56:13,713 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-26 11:56:13,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=236, cellSize=10806, totalHeapSize=24784, min timestamp=1679831773703, max timestamp=1679831773711Num uniques 1086; 2023-03-26 11:56:13,714 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49736, new segment data size=10806, suffix heap size=140912,new segment heap size=24784 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=236 2023-03-26 11:56:13,728 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:13,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:13,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:13,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:13,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:13,731 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:13,731 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,731 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:13,731 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:13,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,733 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,734 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,734 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679831773722, max timestamp=1679831773731Num uniques 1095; 2023-03-26 11:56:13,734 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-26 11:56:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679831773722, max timestamp=1679831773731Num uniques 1095; 2023-03-26 11:56:13,734 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,734 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-26 11:56:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:13,734 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1679831773722, max timestamp=1679831773731Num uniques 1095; 2023-03-26 11:56:13,734 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,735 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50150, new segment data size=8930, suffix heap size=141848,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1095, new segment cells count=195 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:13,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:13,735 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,735 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:13,735 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:13,735 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:13,735 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:13,737 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,737 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,737 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,737 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:13,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:13,739 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc0baee[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:13,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:13,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,740 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,740 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:13,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:13,740 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,741 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:13,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,743 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:13,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:13,746 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:13,746 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:13,746 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:13,746 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:13,746 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:13,748 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,748 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,748 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,748 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,751 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:13,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:13,752 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,752 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,752 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:13,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,752 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:13,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679831773746, max timestamp=1679831773755Num uniques 1054; 2023-03-26 11:56:13,757 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-26 11:56:13,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679831773746, max timestamp=1679831773755Num uniques 1054; 2023-03-26 11:56:13,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1679831773746, max timestamp=1679831773755Num uniques 1054; 2023-03-26 11:56:13,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-26 11:56:13,758 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-03-26 11:56:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679831773771, max timestamp=1679831773776Num uniques 1013; 2023-03-26 11:56:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679831773771, max timestamp=1679831773776Num uniques 1013; 2023-03-26 11:56:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 11:56:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 11:56:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1679831773771, max timestamp=1679831773776Num uniques 1013; 2023-03-26 11:56:13,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-03-26 11:56:13,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679831773784, max timestamp=1679831773795Num uniques 1022; 2023-03-26 11:56:13,797 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-26 11:56:13,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679831773784, max timestamp=1679831773795Num uniques 1022; 2023-03-26 11:56:13,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=272, cellSize=12452, totalHeapSize=28528, min timestamp=1679831773784, max timestamp=1679831773795Num uniques 1022; 2023-03-26 11:56:13,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-26 11:56:13,798 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=12452, suffix heap size=134256,new segment heap size=28528 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=272 2023-03-26 11:56:13,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679831773812, max timestamp=1679831773816Num uniques 1131; 2023-03-26 11:56:13,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679831773812, max timestamp=1679831773816Num uniques 1131; 2023-03-26 11:56:13,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-03-26 11:56:13,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-03-26 11:56:13,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679831773812, max timestamp=1679831773816Num uniques 1131; 2023-03-26 11:56:13,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51796, new segment data size=8286, suffix heap size=145592,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1131, new segment cells count=181 2023-03-26 11:56:13,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1679831773830, max timestamp=1679831773835Num uniques 1040; 2023-03-26 11:56:13,837 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=10980, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-26 11:56:13,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10980, totalHeapSize=25200, min timestamp=1679831773830, max timestamp=1679831773835Num uniques 1040; 2023-03-26 11:56:13,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10982, totalHeapSize=25200, min timestamp=1679831773830, max timestamp=1679831773835Num uniques 1040; 2023-03-26 11:56:13,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47620, new segment data size=10980, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-26 11:56:13,838 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47622, new segment data size=10982, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-03-26 11:56:13,852 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:13,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:13,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:13,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:13,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:13,856 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:13,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:13,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:13,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679831773844, max timestamp=1679831773854Num uniques 1099; 2023-03-26 11:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-26 11:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679831773844, max timestamp=1679831773854Num uniques 1099; 2023-03-26 11:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-26 11:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1679831773844, max timestamp=1679831773854Num uniques 1099; 2023-03-26 11:56:13,857 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-03-26 11:56:13,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:13,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:13,860 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:13,860 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:13,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:13,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:13,861 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,861 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:13,861 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:13,861 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:13,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:13,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,864 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:13,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:13,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bfe858f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:13,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:13,866 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,867 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,867 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:13,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:13,867 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:13,867 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:13,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,869 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:13,869 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,869 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:13,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:13,872 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:13,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:13,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:13,872 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:13,872 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:13,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,875 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,878 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679831773864, max timestamp=1679831773876Num uniques 1108; 2023-03-26 11:56:13,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-03-26 11:56:13,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679831773864, max timestamp=1679831773876Num uniques 1108; 2023-03-26 11:56:13,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-03-26 11:56:13,879 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=258, cellSize=11810, totalHeapSize=27072, min timestamp=1679831773864, max timestamp=1679831773876Num uniques 1108; 2023-03-26 11:56:13,880 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=11810, suffix heap size=143200,new segment heap size=27072 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=258 2023-03-26 11:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,883 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:13,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:13,883 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:13,883 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:13,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:13,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:13,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,918 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831833893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,918 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831833896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9922, totalHeapSize=22808, min timestamp=1679831773891, max timestamp=1679831773898Num uniques 1117; 2023-03-26 11:56:13,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=217, cellSize=9932, totalHeapSize=22808, min timestamp=1679831773891, max timestamp=1679831773898Num uniques 1117; 2023-03-26 11:56:13,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=51142, new segment data size=9922, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-03-26 11:56:13,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=9932, suffix heap size=144136,new segment heap size=22808 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=217 2023-03-26 11:56:13,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1679831773891, max timestamp=1679831773898Num uniques 1117; 2023-03-26 11:56:13,920 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51152, new segment data size=8286, suffix heap size=144136,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1117, new segment cells count=181 2023-03-26 11:56:13,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679831773934, max timestamp=1679831773944Num uniques 1076; 2023-03-26 11:56:13,946 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-03-26 11:56:13,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679831773934, max timestamp=1679831773944Num uniques 1076; 2023-03-26 11:56:13,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=326, cellSize=14926, totalHeapSize=34144, min timestamp=1679831773934, max timestamp=1679831773944Num uniques 1040; 2023-03-26 11:56:13,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49276, new segment data size=14926, suffix heap size=139872,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1076, new segment cells count=326 2023-03-26 11:56:13,947 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=14926, suffix heap size=136128,new segment heap size=34144 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=326 2023-03-26 11:56:13,976 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:13,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831833971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679831773958, max timestamp=1679831773975Num uniques 1185; 2023-03-26 11:56:13,978 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-03-26 11:56:13,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679831773958, max timestamp=1679831773975Num uniques 1185; 2023-03-26 11:56:13,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1679831773958, max timestamp=1679831773975Num uniques 1185; 2023-03-26 11:56:13,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-03-26 11:56:13,980 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=54270, new segment data size=10760, suffix heap size=151208,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1185, new segment cells count=235 2023-03-26 11:56:13,983 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:13,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:13,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:13,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:13,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:13,988 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:13,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:13,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:13,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,988 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,989 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:13,989 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:13,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,990 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:13,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:13,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:13,991 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:13,991 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:13,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:13,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:13,991 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:13,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:13,993 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:13,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:13,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:13,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:13,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:13,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:13,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:13,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c312bc[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:13,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:13,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,005 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:14,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:14,005 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:14,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,006 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:14,007 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326295680c782184dcb90581dcddf61f62d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326295680c782184dcb90581dcddf61f62d 2023-03-26 11:56:14,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:14,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,008 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:14,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:14,011 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:14,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:14,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:14,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:14,011 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:14,012 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=359, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/24b8b494506d4bfe9c27347ac87a06da 2023-03-26 11:56:14,013 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,013 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,013 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,014 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,015 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:14,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:14,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:14,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,019 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:14,019 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:14,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1679831773997, max timestamp=1679831774030Num uniques 1094; 2023-03-26 11:56:14,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-26 11:56:14,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1679831773997, max timestamp=1679831774030Num uniques 1094; 2023-03-26 11:56:14,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-26 11:56:14,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=294, cellSize=13464, totalHeapSize=30816, min timestamp=1679831773997, max timestamp=1679831774030Num uniques 1094; 2023-03-26 11:56:14,033 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=13464, suffix heap size=141744,new segment heap size=30816 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=294 2023-03-26 11:56:14,053 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,053 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831834049, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,053 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831834051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679831774047, max timestamp=1679831774052Num uniques 1153; 2023-03-26 11:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679831774047, max timestamp=1679831774052Num uniques 1153; 2023-03-26 11:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1679831774047, max timestamp=1679831774052Num uniques 1153; 2023-03-26 11:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-03-26 11:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-03-26 11:56:14,055 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52805, new segment data size=9295, suffix heap size=147880,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1153, new segment cells count=203 2023-03-26 11:56:14,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679831774074, max timestamp=1679831774085Num uniques 1062; 2023-03-26 11:56:14,087 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-03-26 11:56:14,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679831774074, max timestamp=1679831774085Num uniques 1062; 2023-03-26 11:56:14,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=212, cellSize=9702, totalHeapSize=22288, min timestamp=1679831774074, max timestamp=1679831774085Num uniques 1062; 2023-03-26 11:56:14,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-03-26 11:56:14,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=9702, suffix heap size=138416,new segment heap size=22288 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=212 2023-03-26 11:56:14,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679831774093, max timestamp=1679831774114Num uniques 1071; 2023-03-26 11:56:14,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-26 11:56:14,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679831774093, max timestamp=1679831774114Num uniques 1071; 2023-03-26 11:56:14,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1679831774093, max timestamp=1679831774114Num uniques 1071; 2023-03-26 11:56:14,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-26 11:56:14,117 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49046, new segment data size=10116, suffix heap size=139352,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1071, new segment cells count=221 2023-03-26 11:56:14,119 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:14,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:14,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:14,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:14,122 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:14,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:14,122 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:14,123 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,123 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,125 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,125 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,125 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,125 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:14,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:14,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:14,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:14,126 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:14,126 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,126 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:14,127 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,127 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:14,127 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:14,127 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:14,127 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:14,129 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,129 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:14,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:14,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ba2c509[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:14,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:14,130 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,132 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:14,132 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:14,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,132 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:14,133 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,133 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:14,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:14,137 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:14,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:14,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:14,137 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:14,137 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:14,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,139 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,140 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,140 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,142 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679831774135, max timestamp=1679831774141Num uniques 1080; 2023-03-26 11:56:14,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-26 11:56:14,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679831774135, max timestamp=1679831774141Num uniques 1080; 2023-03-26 11:56:14,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-26 11:56:14,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:14,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,144 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:14,144 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,145 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:14,145 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:14,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=230, cellSize=10530, totalHeapSize=24160, min timestamp=1679831774135, max timestamp=1679831774141Num uniques 1080; 2023-03-26 11:56:14,146 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=10530, suffix heap size=140288,new segment heap size=24160 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=230 2023-03-26 11:56:14,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:14,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679831774161, max timestamp=1679831774167Num uniques 1089; 2023-03-26 11:56:14,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-26 11:56:14,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=289, cellSize=13234, totalHeapSize=30296, min timestamp=1679831774156, max timestamp=1679831774166Num uniques 1089; 2023-03-26 11:56:14,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=13234, suffix heap size=141224,new segment heap size=30296 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=289 2023-03-26 11:56:14,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=189, cellSize=8654, totalHeapSize=19896, min timestamp=1679831774161, max timestamp=1679831774166Num uniques 1089; 2023-03-26 11:56:14,169 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49874, new segment data size=8654, suffix heap size=141224,new segment heap size=19896 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1089, new segment cells count=189 2023-03-26 11:56:14,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1679831774174, max timestamp=1679831774185Num uniques 1148; 2023-03-26 11:56:14,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52578, new segment data size=15938, suffix heap size=147360,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1148, new segment cells count=348 2023-03-26 11:56:14,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1679831774174, max timestamp=1679831774185Num uniques 1048; 2023-03-26 11:56:14,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=15938, suffix heap size=136960,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=348 2023-03-26 11:56:14,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=348, cellSize=15938, totalHeapSize=36432, min timestamp=1679831774174, max timestamp=1679831774185Num uniques 1048; 2023-03-26 11:56:14,188 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47998, new segment data size=15938, suffix heap size=136960,new segment heap size=36432 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1048, new segment cells count=348 2023-03-26 11:56:14,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831834201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,208 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831834202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831834202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,208 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831834204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,212 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,212 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831834208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,245 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:14,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:14,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:14,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:14,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:14,248 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:14,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:14,248 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:14,249 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,249 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,250 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,250 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,251 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,251 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:14,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:14,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:14,252 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:14,252 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:14,254 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,254 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:14,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72dc1b59[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:14,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:14,255 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,257 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:14,258 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,258 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:14,258 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:14,259 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,259 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:14,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:14,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:14,262 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:14,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:14,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:14,262 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:14,262 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:14,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,264 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,265 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,266 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,268 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,268 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,269 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:14,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,269 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:14,269 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:14,269 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:14,269 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:14,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831834309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831834309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831834309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,312 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831834309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,315 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831834313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,369 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:14,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:14,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:14,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:14,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:14,372 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:14,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:14,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:14,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,373 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,376 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,376 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:14,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,377 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:14,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,377 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:14,377 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:14,377 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:14,377 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:14,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:14,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:14,378 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:14,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:14,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:14,378 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:14,379 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,379 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,380 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:14,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5da13a70[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:14,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:14,382 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,383 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:14,383 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:14,383 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,384 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:14,385 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:14,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,385 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:14,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:14,387 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:14,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:14,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:14,387 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:14,387 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:14,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,393 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,394 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,395 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,397 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:14,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:14,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:14,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:14,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:14,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,399 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:14,399 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:14,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/8899de74748847348bb5170f49aef8e7 2023-03-26 11:56:14,430 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=359 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/a21b90734b654adcbb452fe9caaaf574 2023-03-26 11:56:14,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/24b8b494506d4bfe9c27347ac87a06da as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/24b8b494506d4bfe9c27347ac87a06da 2023-03-26 11:56:14,437 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/24b8b494506d4bfe9c27347ac87a06da, entries=150, sequenceid=359, filesize=25.4 K 2023-03-26 11:56:14,438 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/8899de74748847348bb5170f49aef8e7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8899de74748847348bb5170f49aef8e7 2023-03-26 11:56:14,441 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8899de74748847348bb5170f49aef8e7, entries=150, sequenceid=359, filesize=11.9 K 2023-03-26 11:56:14,442 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/a21b90734b654adcbb452fe9caaaf574 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a21b90734b654adcbb452fe9caaaf574 2023-03-26 11:56:14,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a21b90734b654adcbb452fe9caaaf574, entries=150, sequenceid=359, filesize=11.9 K 2023-03-26 11:56:14,446 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~65.34 KB/66912, currentSize=161.02 KB/164880 for 5890b64a473c744d0cc3b74eead11b50 in 857ms, sequenceid=359, compaction requested=true 2023-03-26 11:56:14,446 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:14,446 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:14,446 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:14,446 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:14,447 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:14,447 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:14,447 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49460 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:14,447 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,447 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:14,447 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,447 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b05b4b46bdea42369b1c581f7dd07df8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0cbc503d1a6c4c38b42327c2313f857e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c113121e841e44e89885641e37b9f7a5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/24b8b494506d4bfe9c27347ac87a06da] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=102.4 K 2023-03-26 11:56:14,447 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/12d9e1c0b50648fb9ff1a080050e1446, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7fe857a540e64a62abe4213442c9c0fb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/dad75ada66c3412fb04ca39c00aefc83, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8899de74748847348bb5170f49aef8e7] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=48.3 K 2023-03-26 11:56:14,448 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 12d9e1c0b50648fb9ff1a080050e1446, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679831771448 2023-03-26 11:56:14,448 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b05b4b46bdea42369b1c581f7dd07df8, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679831771448 2023-03-26 11:56:14,448 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 7fe857a540e64a62abe4213442c9c0fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679831771488 2023-03-26 11:56:14,450 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 0cbc503d1a6c4c38b42327c2313f857e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679831771488 2023-03-26 11:56:14,450 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting dad75ada66c3412fb04ca39c00aefc83, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679831772088 2023-03-26 11:56:14,450 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting c113121e841e44e89885641e37b9f7a5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679831772088 2023-03-26 11:56:14,450 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8899de74748847348bb5170f49aef8e7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679831773274 2023-03-26 11:56:14,451 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 24b8b494506d4bfe9c27347ac87a06da, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679831773274 2023-03-26 11:56:14,458 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#570 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:14,474 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#571 average throughput is 1.52 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:14,499 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:14,499 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:14,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:14,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:14,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:14,502 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:14,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:14,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:14,502 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,503 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,504 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:14,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:14,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,504 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:14,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:14,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:14,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:14,506 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:14,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:14,506 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:14,506 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,506 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:14,506 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=161.02 KB heapSize=446.95 KB 2023-03-26 11:56:14,507 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:14,507 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:14,507 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:14,507 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:14,507 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:14,507 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:14,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:14,515 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:14,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831834516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,519 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831834516, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,522 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831834518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,523 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831834519, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831834520, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,530 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303261e6d201759384238950c86714411859e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261e6d201759384238950c86714411859e 2023-03-26 11:56:14,541 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=757, memsize=53.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/a130bf9b6796476b8e893ddb9c0b1448 2023-03-26 11:56:14,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831834620, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831834623, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831834624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,628 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831834625, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,824 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831834821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831834825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831834826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831834829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,830 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:14,830 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831834830, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,883 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/30dbcba9bcbf46ffa190217ce467f333 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/30dbcba9bcbf46ffa190217ce467f333 2023-03-26 11:56:14,889 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 30dbcba9bcbf46ffa190217ce467f333(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:14,889 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:14,889 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=12, startTime=1679831774446; duration=0sec 2023-03-26 11:56:14,889 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:14,889 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-03-26 11:56:14,891 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61444 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-03-26 11:56:14,891 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:14,891 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,891 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68ff5dc54ef14f7bbf4060b67f1ac9a9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a65101d67253470fadecf451c75092fb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d9b831d328e54cf784913467e6c6e215, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/7c5b61063652463692c0e12b93202a84, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a21b90734b654adcbb452fe9caaaf574] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=60.0 K 2023-03-26 11:56:14,892 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 68ff5dc54ef14f7bbf4060b67f1ac9a9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=258, earliestPutTs=1679831770811 2023-03-26 11:56:14,892 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a65101d67253470fadecf451c75092fb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=269, earliestPutTs=1679831771448 2023-03-26 11:56:14,893 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d9b831d328e54cf784913467e6c6e215, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=307, earliestPutTs=1679831771488 2023-03-26 11:56:14,893 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 7c5b61063652463692c0e12b93202a84, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=321, earliestPutTs=1679831772088 2023-03-26 11:56:14,893 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a21b90734b654adcbb452fe9caaaf574, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679831773274 2023-03-26 11:56:14,894 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/96365ea8f4564857977a0ef1e165a076 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96365ea8f4564857977a0ef1e165a076 2023-03-26 11:56:14,898 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 96365ea8f4564857977a0ef1e165a076(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:14,898 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:14,898 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=12, startTime=1679831774446; duration=0sec 2023-03-26 11:56:14,898 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:14,900 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#574 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:14,964 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/9f5049cfca4a48929b527d4b3b866f1a 2023-03-26 11:56:14,973 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=53.67 KB at sequenceid=757 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/11a0587e2ef44c29aebb79ecca4349e0 2023-03-26 11:56:14,977 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/a130bf9b6796476b8e893ddb9c0b1448 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a130bf9b6796476b8e893ddb9c0b1448 2023-03-26 11:56:14,980 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a130bf9b6796476b8e893ddb9c0b1448, entries=150, sequenceid=757, filesize=25.4 K 2023-03-26 11:56:14,980 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/9f5049cfca4a48929b527d4b3b866f1a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9f5049cfca4a48929b527d4b3b866f1a 2023-03-26 11:56:14,984 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9f5049cfca4a48929b527d4b3b866f1a, entries=150, sequenceid=757, filesize=11.9 K 2023-03-26 11:56:14,985 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/11a0587e2ef44c29aebb79ecca4349e0 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/11a0587e2ef44c29aebb79ecca4349e0 2023-03-26 11:56:14,988 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/11a0587e2ef44c29aebb79ecca4349e0, entries=150, sequenceid=757, filesize=11.9 K 2023-03-26 11:56:14,989 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~161.02 KB/164880, heapSize ~446.91 KB/457632, currentSize=26.84 KB/27480 for 5890b64a473c744d0cc3b74eead11b50 in 483ms, sequenceid=757, compaction requested=false 2023-03-26 11:56:14,989 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:14,989 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:14,989 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,991 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:14,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@466c858a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:14,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:14,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,995 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:14,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:14,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:14,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:14,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:14,998 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:14,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:14,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:14,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:15,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,001 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,001 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,005 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,007 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,008 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,008 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,008 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,008 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,008 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,008 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,008 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,008 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,008 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,008 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,108 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:15,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:15,111 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:15,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:15,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:15,112 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:15,112 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,112 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:15,112 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:15,112 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,112 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,114 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:15,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,115 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,115 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:15,115 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:15,115 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:15,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:15,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,115 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:15,116 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:15,116 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:15,116 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,130 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326aa884b10e30246debf75f1a8783dbc1a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326aa884b10e30246debf75f1a8783dbc1a 2023-03-26 11:56:15,137 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=766, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/c9b3af229bf147e98a8e7975deaa1ace 2023-03-26 11:56:15,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:15,139 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:15,147 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/9e80e8f78a4e4981932703852a98b27f 2023-03-26 11:56:15,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831775148, max timestamp=1679831775156Num uniques 859; 2023-03-26 11:56:15,159 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:15,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831775148, max timestamp=1679831775156Num uniques 859; 2023-03-26 11:56:15,161 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:15,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1679831775148, max timestamp=1679831775156Num uniques 859; 2023-03-26 11:56:15,162 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-03-26 11:56:15,163 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=766 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/ffdf9502eefe49e4a3074c8e1e28b434 2023-03-26 11:56:15,167 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/c9b3af229bf147e98a8e7975deaa1ace as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c9b3af229bf147e98a8e7975deaa1ace 2023-03-26 11:56:15,170 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c9b3af229bf147e98a8e7975deaa1ace, entries=100, sequenceid=766, filesize=18.6 K 2023-03-26 11:56:15,171 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/9e80e8f78a4e4981932703852a98b27f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9e80e8f78a4e4981932703852a98b27f 2023-03-26 11:56:15,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679831775167, max timestamp=1679831775174Num uniques 1009; 2023-03-26 11:56:15,176 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-03-26 11:56:15,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679831775167, max timestamp=1679831775174Num uniques 1068; 2023-03-26 11:56:15,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 11:56:15,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1679831775167, max timestamp=1679831775174Num uniques 1068; 2023-03-26 11:56:15,177 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=7688, suffix heap size=139040,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=168 2023-03-26 11:56:15,180 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9e80e8f78a4e4981932703852a98b27f, entries=100, sequenceid=766, filesize=9.5 K 2023-03-26 11:56:15,181 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/ffdf9502eefe49e4a3074c8e1e28b434 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/ffdf9502eefe49e4a3074c8e1e28b434 2023-03-26 11:56:15,185 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/ffdf9502eefe49e4a3074c8e1e28b434, entries=100, sequenceid=766, filesize=9.5 K 2023-03-26 11:56:15,185 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=80.51 KB/82440 for 5890b64a473c744d0cc3b74eead11b50 in 69ms, sequenceid=766, compaction requested=true 2023-03-26 11:56:15,185 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:15,186 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:15,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:15,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:15,186 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:15,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:15,188 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,188 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=93.93 KB heapSize=264.61 KB 2023-03-26 11:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:15,189 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:15,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,189 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:15,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:15,192 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4beb8ac3[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:15,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:15,192 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:15,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:15,194 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:15,196 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:15,196 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,196 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:15,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,196 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:15,197 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,197 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,197 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,198 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,199 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,199 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,200 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,203 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,204 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,204 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,204 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:15,204 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,204 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,212 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032653a2519c2e69441287fb307f25e0b987 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032653a2519c2e69441287fb307f25e0b987 2023-03-26 11:56:15,215 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 290 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831835208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,219 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831835214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831835215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,221 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831835215, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,228 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=815, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/fa26320c60c841039bddee0a1abe668d 2023-03-26 11:56:15,248 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/8a937e5af1864dbd9e2d09dedefa5256 2023-03-26 11:56:15,304 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:15,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:15,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:15,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:15,309 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:15,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:15,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:15,309 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/1ae46615dec74091a71b359784de9eb2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/1ae46615dec74091a71b359784de9eb2 2023-03-26 11:56:15,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,310 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,312 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:15,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,312 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,313 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:15,313 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:15,313 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,313 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:15,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:15,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:15,314 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:15,314 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:15,315 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 1ae46615dec74091a71b359784de9eb2(size=12.7 K), total size for store is 34.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:15,315 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:15,315 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=11, startTime=1679831774446; duration=0sec 2023-03-26 11:56:15,315 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:15,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,317 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:15,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,319 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,319 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 292 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831835316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:15,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@597f5634[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:15,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:15,319 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,321 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:15,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,321 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:15,321 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:15,322 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831835320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,323 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831835322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,324 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831835322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,324 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,324 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:15,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,325 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,326 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831835326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:15,327 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,327 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,327 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,328 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,328 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,328 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,330 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,330 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,331 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,332 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,337 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,338 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,337 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,338 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,338 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:15,338 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,338 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,438 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:15,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:15,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:15,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:15,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:15,447 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:15,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:15,447 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:15,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,448 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,450 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,450 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:15,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,451 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:15,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:15,451 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:15,451 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:15,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:15,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:15,452 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:15,452 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:15,454 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,454 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:15,456 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd985d1[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:15,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:15,456 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,457 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:15,457 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:15,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,457 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:15,458 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:15,458 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,459 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:15,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,461 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,461 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,461 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,464 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,465 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,466 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,469 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,469 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,470 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,470 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,470 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,470 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,523 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 294 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831835521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,525 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831835523, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,527 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831835525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831835525, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,570 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:15,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:15,572 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:15,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:15,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:15,573 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:15,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:15,573 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:15,574 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,574 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,575 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,575 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,576 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:15,576 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:15,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,576 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:15,577 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:15,577 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:15,577 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,577 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:15,577 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:15,577 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:15,577 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:15,580 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,580 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,581 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:15,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fbb1135[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:15,583 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:15,583 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,585 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,585 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,585 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:15,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:15,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:15,585 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:15,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:15,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,587 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,587 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:15,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,589 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,590 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,590 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,590 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,590 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,591 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,591 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,591 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,592 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,594 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,599 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,599 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,600 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,599 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,600 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,600 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=815 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/8526df90c0154f0ebe3a247bdc49a37f 2023-03-26 11:56:15,681 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/fa26320c60c841039bddee0a1abe668d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/fa26320c60c841039bddee0a1abe668d 2023-03-26 11:56:15,684 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/fa26320c60c841039bddee0a1abe668d, entries=200, sequenceid=815, filesize=32.3 K 2023-03-26 11:56:15,685 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/8a937e5af1864dbd9e2d09dedefa5256 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8a937e5af1864dbd9e2d09dedefa5256 2023-03-26 11:56:15,696 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8a937e5af1864dbd9e2d09dedefa5256, entries=150, sequenceid=815, filesize=11.9 K 2023-03-26 11:56:15,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/8526df90c0154f0ebe3a247bdc49a37f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8526df90c0154f0ebe3a247bdc49a37f 2023-03-26 11:56:15,700 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:15,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:15,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8526df90c0154f0ebe3a247bdc49a37f, entries=150, sequenceid=815, filesize=11.9 K 2023-03-26 11:56:15,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~284.48 KB/291312, currentSize=80.51 KB/82440 for 5890b64a473c744d0cc3b74eead11b50 in 514ms, sequenceid=815, compaction requested=true 2023-03-26 11:56:15,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:15,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:15,702 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:15,702 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:15,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:15,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:15,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:15,705 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 105028 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:15,705 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:15,705 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,705 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96365ea8f4564857977a0ef1e165a076, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a130bf9b6796476b8e893ddb9c0b1448, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c9b3af229bf147e98a8e7975deaa1ace, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/fa26320c60c841039bddee0a1abe668d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=102.6 K 2023-03-26 11:56:15,706 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 96365ea8f4564857977a0ef1e165a076, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679831773274 2023-03-26 11:56:15,706 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47152 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:15,706 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting a130bf9b6796476b8e893ddb9c0b1448, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=757, earliestPutTs=1679831774201 2023-03-26 11:56:15,706 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:15,706 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,706 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/30dbcba9bcbf46ffa190217ce467f333, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9f5049cfca4a48929b527d4b3b866f1a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9e80e8f78a4e4981932703852a98b27f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8a937e5af1864dbd9e2d09dedefa5256] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=46.0 K 2023-03-26 11:56:15,706 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c9b3af229bf147e98a8e7975deaa1ace, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1679831774517 2023-03-26 11:56:15,707 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 30dbcba9bcbf46ffa190217ce467f333, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679831773274 2023-03-26 11:56:15,707 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fa26320c60c841039bddee0a1abe668d, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1679831775183 2023-03-26 11:56:15,707 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9f5049cfca4a48929b527d4b3b866f1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=757, earliestPutTs=1679831774201 2023-03-26 11:56:15,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:15,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:15,707 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:15,707 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,707 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 9e80e8f78a4e4981932703852a98b27f, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1679831774517 2023-03-26 11:56:15,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:15,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:15,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,708 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,708 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8a937e5af1864dbd9e2d09dedefa5256, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1679831775183 2023-03-26 11:56:15,710 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,710 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,711 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:15,711 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,712 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:15,712 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:15,712 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,712 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:15,713 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:15,713 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,713 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:15,713 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 11:56:15,713 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:15,713 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,714 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:15,714 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,714 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:15,714 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,715 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#582 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:15,740 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#583 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:15,757 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/73e0f81aad4c4ff58cef60d4a44cff99 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/73e0f81aad4c4ff58cef60d4a44cff99 2023-03-26 11:56:15,761 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326b8bcc14a87a44012a46046fe6381bee6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b8bcc14a87a44012a46046fe6381bee6 2023-03-26 11:56:15,765 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 73e0f81aad4c4ff58cef60d4a44cff99(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:15,765 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:15,765 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=12, startTime=1679831775702; duration=0sec 2023-03-26 11:56:15,766 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:15,766 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:15,767 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47084 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:15,767 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:15,767 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,767 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/1ae46615dec74091a71b359784de9eb2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/11a0587e2ef44c29aebb79ecca4349e0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/ffdf9502eefe49e4a3074c8e1e28b434, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8526df90c0154f0ebe3a247bdc49a37f] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=46.0 K 2023-03-26 11:56:15,767 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 1ae46615dec74091a71b359784de9eb2, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=359, earliestPutTs=1679831773274 2023-03-26 11:56:15,768 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 11a0587e2ef44c29aebb79ecca4349e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=757, earliestPutTs=1679831774201 2023-03-26 11:56:15,768 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ffdf9502eefe49e4a3074c8e1e28b434, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=766, earliestPutTs=1679831774517 2023-03-26 11:56:15,768 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 8526df90c0154f0ebe3a247bdc49a37f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1679831775183 2023-03-26 11:56:15,770 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=831, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/3def535288cf41a1a2a3be467933dfd9 2023-03-26 11:56:15,782 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#585 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:15,808 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/9184ee8dc8ef43dca8fd5f9f3adee32b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9184ee8dc8ef43dca8fd5f9f3adee32b 2023-03-26 11:56:15,810 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/e0d160dfa1e94c8483c3a04a6bd4ee05 2023-03-26 11:56:15,813 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 9184ee8dc8ef43dca8fd5f9f3adee32b(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:15,813 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:15,813 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=12, startTime=1679831775702; duration=0sec 2023-03-26 11:56:15,813 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:15,821 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=831 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/3d88f71853da400594d89d304878ff2d 2023-03-26 11:56:15,826 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/3def535288cf41a1a2a3be467933dfd9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3def535288cf41a1a2a3be467933dfd9 2023-03-26 11:56:15,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:15,827 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:15,831 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3def535288cf41a1a2a3be467933dfd9, entries=150, sequenceid=831, filesize=25.4 K 2023-03-26 11:56:15,831 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/e0d160dfa1e94c8483c3a04a6bd4ee05 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e0d160dfa1e94c8483c3a04a6bd4ee05 2023-03-26 11:56:15,835 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e0d160dfa1e94c8483c3a04a6bd4ee05, entries=150, sequenceid=831, filesize=11.9 K 2023-03-26 11:56:15,836 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/3d88f71853da400594d89d304878ff2d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3d88f71853da400594d89d304878ff2d 2023-03-26 11:56:15,840 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3d88f71853da400594d89d304878ff2d, entries=150, sequenceid=831, filesize=11.9 K 2023-03-26 11:56:15,840 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=46.96 KB/48090 for 5890b64a473c744d0cc3b74eead11b50 in 127ms, sequenceid=831, compaction requested=false 2023-03-26 11:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:15,840 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,840 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:15,841 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:15,842 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,842 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,843 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:15,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:15,844 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:56:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:15,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,844 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:15,844 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:15,845 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:15,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:15,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38113b93[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:15,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:15,845 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,847 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:15,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:15,847 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:15,847 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,847 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:15,848 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:15,848 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,849 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:15,852 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,852 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,852 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,852 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,852 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,852 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,855 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,858 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,864 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,864 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,864 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,865 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,865 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,865 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,865 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,865 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831835865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831835866, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,870 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831835868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831775861, max timestamp=1679831775868Num uniques 859; 2023-03-26 11:56:15,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:15,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1679831775861, max timestamp=1679831775868Num uniques 859; 2023-03-26 11:56:15,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-03-26 11:56:15,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831775861, max timestamp=1679831775868Num uniques 859; 2023-03-26 11:56:15,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:15,925 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831835922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,965 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:15,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:15,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:15,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:15,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:15,968 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:15,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:15,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:15,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,968 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,970 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,970 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:15,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,971 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:15,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,971 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:15,972 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:15,972 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:15,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:15,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:15,972 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:15,972 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:15,973 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:15,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831835971, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,973 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 265 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831835972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,975 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,975 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,975 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:15,975 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831835972, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:15,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bfc0453[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:15,977 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:15,977 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,978 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,978 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:15,978 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:15,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:15,979 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,980 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,980 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:15,983 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:15,983 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:15,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:15,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:15,983 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:15,983 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:15,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:15,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,985 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:15,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:15,986 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:15,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,987 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:15,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:15,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:15,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:15,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:15,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:15,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:15,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:15,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,991 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:15,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:15,991 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:15,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:15,991 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:15,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:15,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,027 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 293 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831836026, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,091 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,094 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,094 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,095 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,096 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,096 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,097 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,097 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,098 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,098 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,098 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,098 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,098 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,098 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,098 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,098 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,100 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,100 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5acafdf0[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,102 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,102 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,104 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,104 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,104 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,104 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,107 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,107 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,109 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,110 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,110 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,112 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,113 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,114 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,116 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,116 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,117 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,116 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,117 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831836175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 267 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831836175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831836177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,217 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,217 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,219 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,220 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,220 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,221 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,222 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,222 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,223 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,223 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,224 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,224 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,224 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,224 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,224 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,226 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,226 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,228 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0f6d25[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,228 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,228 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,229 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,229 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,230 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,230 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,230 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 295 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831836230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,232 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,232 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,233 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,235 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,235 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,235 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,237 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,238 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,239 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,242 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:16,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,242 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,242 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,243 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,258 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303269caca11d1df44beb982c870a5768ded1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269caca11d1df44beb982c870a5768ded1 2023-03-26 11:56:16,334 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831836332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,343 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,346 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,346 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,346 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,346 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,346 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,347 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,348 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,348 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,349 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,349 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,349 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,350 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,350 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,350 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,350 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,350 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,354 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,355 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,356 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c7306b4[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,356 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,358 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,359 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,359 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,360 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,360 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,361 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,363 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,363 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,363 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,363 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,363 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,365 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,365 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,365 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,366 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,367 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,370 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,371 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,371 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,371 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,371 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:16,472 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,473 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,475 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,475 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,477 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,477 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,478 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,478 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,479 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,479 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,479 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,479 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,479 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,480 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,481 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,481 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,482 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@675657f5[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,483 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,483 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831836480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831836482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,484 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 269 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831836482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,487 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,488 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,488 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,489 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,489 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,491 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,491 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,491 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,495 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,495 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,495 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,495 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,496 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,499 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,499 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:16,499 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,499 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,535 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 297 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831836534, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,577 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/79f3fefa97d944e980a28860e168a733 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/79f3fefa97d944e980a28860e168a733 2023-03-26 11:56:16,582 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 79f3fefa97d944e980a28860e168a733(size=26.4 K), total size for store is 51.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:16,582 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:16,582 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=12, startTime=1679831775702; duration=0sec 2023-03-26 11:56:16,582 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:16,599 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,603 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,605 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,605 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,607 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,607 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,607 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,607 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,609 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,609 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,609 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,609 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6afe0392[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,611 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,611 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,613 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,613 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,613 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,613 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,614 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,614 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,615 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,621 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,621 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,621 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,621 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,623 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,624 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,625 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,628 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,628 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,628 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:16,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,628 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,628 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,664 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=844, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/e5ae6a7d5e254b1784488e125961bfa2 2023-03-26 11:56:16,728 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,730 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,732 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,732 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,738 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,739 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,739 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,740 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,740 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,740 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,740 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,740 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,740 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,740 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,741 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,741 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,741 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,741 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,743 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,747 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,749 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cce3dad[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,749 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,752 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,753 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,753 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,754 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,755 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,757 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,757 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,760 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,761 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,763 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,764 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,764 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,764 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:16,764 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,764 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,864 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,866 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,867 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,868 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,869 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,869 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,870 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,871 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,871 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,871 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,871 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,871 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,872 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,874 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,874 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:16,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:16,876 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@388cca26[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:16,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:16,876 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,877 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:16,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:16,877 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:16,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,877 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:16,879 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,879 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:16,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:16,881 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:16,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:16,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:16,881 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:16,881 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:16,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,883 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:16,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:16,884 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,885 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:16,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,888 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:16,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:16,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,888 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:16,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:16,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:16,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:16,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:16,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,889 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:16,889 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:16,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,889 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,989 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831836986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,989 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:16,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:16,991 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 271 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831836989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:16,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:16,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:16,992 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:16,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:16,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:16,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,993 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,993 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:16,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831836991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,994 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:16,994 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:16,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:16,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,995 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:16,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:16,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:16,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:16,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:16,996 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:16,996 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:16,999 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:16,999 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,000 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:17,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:17,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6db0cb3d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:17,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:17,001 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,003 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:17,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:17,003 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:17,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,003 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:17,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:17,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:17,006 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:17,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:17,009 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:17,009 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:17,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:17,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:17,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:17,009 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:17,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:17,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:17,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:17,012 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:17,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:17,014 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:17,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:17,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:17,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:17,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:17,018 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:17,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:17,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:17,018 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:17,018 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:17,018 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,018 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:17,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:17,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:17,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:17,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:17,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:17,019 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,039 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:17,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831837037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,075 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/570ed5fbbaec4b2cb66c2dc92873c390 2023-03-26 11:56:17,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=844 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/219712a3fca745769910643259be0b22 2023-03-26 11:56:17,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/e5ae6a7d5e254b1784488e125961bfa2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/e5ae6a7d5e254b1784488e125961bfa2 2023-03-26 11:56:17,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/e5ae6a7d5e254b1784488e125961bfa2, entries=300, sequenceid=844, filesize=46.1 K 2023-03-26 11:56:17,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/570ed5fbbaec4b2cb66c2dc92873c390 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/570ed5fbbaec4b2cb66c2dc92873c390 2023-03-26 11:56:17,096 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/570ed5fbbaec4b2cb66c2dc92873c390, entries=100, sequenceid=844, filesize=9.5 K 2023-03-26 11:56:17,097 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/219712a3fca745769910643259be0b22 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/219712a3fca745769910643259be0b22 2023-03-26 11:56:17,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/219712a3fca745769910643259be0b22, entries=100, sequenceid=844, filesize=9.5 K 2023-03-26 11:56:17,100 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=125.23 KB/128240 for 5890b64a473c744d0cc3b74eead11b50 in 1256ms, sequenceid=844, compaction requested=true 2023-03-26 11:56:17,100 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:17,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:17,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:17,100 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:17,100 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:17,100 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:17,101 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 100246 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:17,101 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:17,101 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35102 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:17,101 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:17,101 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:17,101 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/79f3fefa97d944e980a28860e168a733, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3def535288cf41a1a2a3be467933dfd9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/e5ae6a7d5e254b1784488e125961bfa2] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=97.9 K 2023-03-26 11:56:17,101 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:17,102 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/73e0f81aad4c4ff58cef60d4a44cff99, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e0d160dfa1e94c8483c3a04a6bd4ee05, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/570ed5fbbaec4b2cb66c2dc92873c390] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=34.3 K 2023-03-26 11:56:17,102 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 79f3fefa97d944e980a28860e168a733, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1679831775183 2023-03-26 11:56:17,102 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 73e0f81aad4c4ff58cef60d4a44cff99, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1679831775183 2023-03-26 11:56:17,103 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 3def535288cf41a1a2a3be467933dfd9, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1679831775206 2023-03-26 11:56:17,103 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e0d160dfa1e94c8483c3a04a6bd4ee05, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1679831775206 2023-03-26 11:56:17,103 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting e5ae6a7d5e254b1784488e125961bfa2, keycount=300, bloomtype=ROW, size=46.1 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1679831775826 2023-03-26 11:56:17,103 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 570ed5fbbaec4b2cb66c2dc92873c390, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1679831775838 2023-03-26 11:56:17,108 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#591 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:17,118 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#592 average throughput is 2.46 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:17,119 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:17,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:17,122 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/05b2d1ace6f84a64a42be522a22ad206 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/05b2d1ace6f84a64a42be522a22ad206 2023-03-26 11:56:17,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:17,126 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 05b2d1ace6f84a64a42be522a22ad206(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:17,126 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:17,126 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831777100; duration=0sec 2023-03-26 11:56:17,126 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:17,126 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:17,127 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35034 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:17,127 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:17,127 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:17,127 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9184ee8dc8ef43dca8fd5f9f3adee32b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3d88f71853da400594d89d304878ff2d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/219712a3fca745769910643259be0b22] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=34.2 K 2023-03-26 11:56:17,127 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 9184ee8dc8ef43dca8fd5f9f3adee32b, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=815, earliestPutTs=1679831775183 2023-03-26 11:56:17,128 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3d88f71853da400594d89d304878ff2d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=831, earliestPutTs=1679831775206 2023-03-26 11:56:17,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:17,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:17,128 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:17,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:17,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:17,128 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 219712a3fca745769910643259be0b22, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1679831775838 2023-03-26 11:56:17,128 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:17,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,129 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:17,130 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:17,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:17,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:17,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:17,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:17,131 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:17,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:17,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:17,136 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:17,136 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:17,136 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:17,137 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#593 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:17,140 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:17,140 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:17,140 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:17,140 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=125.23 KB heapSize=355.30 KB 2023-03-26 11:56:17,141 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:17,141 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:17,141 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:17,141 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:17,141 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:17,141 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:17,160 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/8861e1bbe347413c8b47252a41f15a2e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8861e1bbe347413c8b47252a41f15a2e 2023-03-26 11:56:17,165 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 8861e1bbe347413c8b47252a41f15a2e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:17,165 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:17,166 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=13, startTime=1679831777100; duration=0sec 2023-03-26 11:56:17,166 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:17,173 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/97ef788bf688430c9b8736bf5a0e3431 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ef788bf688430c9b8736bf5a0e3431 2023-03-26 11:56:17,177 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 97ef788bf688430c9b8736bf5a0e3431(size=26.5 K), total size for store is 26.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:17,177 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:17,177 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831777100; duration=0sec 2023-03-26 11:56:17,177 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:17,573 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326b6e0722e7743428da1f1125e0bbc26e6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b6e0722e7743428da1f1125e0bbc26e6 2023-03-26 11:56:17,978 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=881, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/af37598183504812aca0495a713c7a4e 2023-03-26 11:56:17,988 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/e29e388e8f9b41678e051ecd19ce829c 2023-03-26 11:56:17,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:17,996 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:18,009 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831838006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831838008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,011 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831838008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,042 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831838041, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,111 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831838110, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831838112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,114 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831838112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,314 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831838313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831838315, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,318 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831838316, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,352 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831838350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,353 DEBUG [Thread-3664] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4144 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:56:18,399 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=881 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/6d72fc7536d84ff1b6b43c3e7b54dfb9 2023-03-26 11:56:18,403 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/af37598183504812aca0495a713c7a4e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/af37598183504812aca0495a713c7a4e 2023-03-26 11:56:18,406 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/af37598183504812aca0495a713c7a4e, entries=150, sequenceid=881, filesize=25.4 K 2023-03-26 11:56:18,407 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/e29e388e8f9b41678e051ecd19ce829c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e29e388e8f9b41678e051ecd19ce829c 2023-03-26 11:56:18,411 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e29e388e8f9b41678e051ecd19ce829c, entries=150, sequenceid=881, filesize=11.9 K 2023-03-26 11:56:18,411 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/6d72fc7536d84ff1b6b43c3e7b54dfb9 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/6d72fc7536d84ff1b6b43c3e7b54dfb9 2023-03-26 11:56:18,414 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/6d72fc7536d84ff1b6b43c3e7b54dfb9, entries=150, sequenceid=881, filesize=11.9 K 2023-03-26 11:56:18,415 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~125.23 KB/128240, heapSize ~355.25 KB/363776, currentSize=53.67 KB/54960 for 5890b64a473c744d0cc3b74eead11b50 in 1275ms, sequenceid=881, compaction requested=false 2023-03-26 11:56:18,415 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:18,415 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:18,415 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:18,417 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,417 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,418 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:18,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7748b78a[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:18,419 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:18,419 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,421 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:18,421 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:18,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,421 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:18,430 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,430 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:18,432 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:18,432 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:18,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:18,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:18,432 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:18,432 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:18,435 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,435 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,435 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,437 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,440 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:18,440 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:18,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,441 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:18,441 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:18,441 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:18,541 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:18,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:18,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:18,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:18,545 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:18,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:18,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:18,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,548 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,548 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,549 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,549 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:18,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:18,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:18,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:18,550 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:18,550 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,550 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:18,550 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-03-26 11:56:18,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:18,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:18,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:18,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:18,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:18,551 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:18,564 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326edf3f0ec7bd64b5e97fd03e46feced6c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326edf3f0ec7bd64b5e97fd03e46feced6c 2023-03-26 11:56:18,569 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=894, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/36f14378cbfc4d68b7cd267c9fb8f9bb 2023-03-26 11:56:18,577 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=894 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/041f0f64ff5e4aa7989f2e62b083f0c1 2023-03-26 11:56:18,585 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=894 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c90b4da55e024e70a10681db77c74f28 2023-03-26 11:56:18,588 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/36f14378cbfc4d68b7cd267c9fb8f9bb as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/36f14378cbfc4d68b7cd267c9fb8f9bb 2023-03-26 11:56:18,591 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/36f14378cbfc4d68b7cd267c9fb8f9bb, entries=150, sequenceid=894, filesize=25.4 K 2023-03-26 11:56:18,592 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/041f0f64ff5e4aa7989f2e62b083f0c1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/041f0f64ff5e4aa7989f2e62b083f0c1 2023-03-26 11:56:18,595 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/041f0f64ff5e4aa7989f2e62b083f0c1, entries=150, sequenceid=894, filesize=11.9 K 2023-03-26 11:56:18,595 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c90b4da55e024e70a10681db77c74f28 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c90b4da55e024e70a10681db77c74f28 2023-03-26 11:56:18,598 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c90b4da55e024e70a10681db77c74f28, entries=150, sequenceid=894, filesize=11.9 K 2023-03-26 11:56:18,599 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 5890b64a473c744d0cc3b74eead11b50 in 49ms, sequenceid=894, compaction requested=true 2023-03-26 11:56:18,599 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:18,599 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:18,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:18,599 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:18,600 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:18,601 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,601 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,602 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,602 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:18,603 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3aa6cf9b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:18,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:18,603 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,605 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:18,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:18,606 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:18,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,606 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:18,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:18,607 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,607 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,608 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:18,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:18,610 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:18,610 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:18,610 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:18,610 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:18,610 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:18,612 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,612 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,612 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,612 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,614 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,616 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:18,616 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,616 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,617 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:18,617 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:18,617 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:18,647 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-03-26 11:56:18,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:18,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:18,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:18,648 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:18,648 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:18,649 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:18,662 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326b5a903f0eef24c3da0e65925ce060c31 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b5a903f0eef24c3da0e65925ce060c31 2023-03-26 11:56:18,668 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=905, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/a822dab9cc894e85ae8e433a163d2ebe 2023-03-26 11:56:18,677 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=905 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/f75df679ddc54ef1b4a3ff05d28a9cc1 2023-03-26 11:56:18,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831838678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831778669, max timestamp=1679831778680Num uniques 859; 2023-03-26 11:56:18,682 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:18,682 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831838679, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831778669, max timestamp=1679831778680Num uniques 859; 2023-03-26 11:56:18,682 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:18,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1679831778669, max timestamp=1679831778680Num uniques 859; 2023-03-26 11:56:18,684 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-03-26 11:56:18,717 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:18,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:18,719 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:18,720 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:18,720 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:18,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:18,720 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:18,721 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,721 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,723 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,723 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,723 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,724 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:18,724 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:18,724 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:18,724 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,724 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:18,724 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:18,724 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,724 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:18,726 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,726 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,727 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:18,728 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64a5ff13[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:18,728 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:18,728 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,729 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,729 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:18,729 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:18,729 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:18,730 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,730 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:18,733 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,733 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:18,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:18,735 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:18,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:18,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:18,735 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:18,735 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:18,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,738 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,739 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,740 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,742 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,743 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:18,743 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,743 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,743 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:18,743 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:18,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831838757, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831838783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,785 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,785 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831838783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,843 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:18,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:18,846 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:18,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:18,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:18,848 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:18,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:18,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:18,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,848 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,850 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,850 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,850 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,850 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:18,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:18,852 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:18,852 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,852 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:18,852 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:18,852 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:18,853 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:18,854 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,855 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,855 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:18,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@fa73999[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:18,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:18,856 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,859 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:18,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:18,859 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:18,860 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,861 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:18,861 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:18,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831838862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:18,864 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:18,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:18,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:18,864 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:18,864 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:18,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,866 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,867 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,869 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:18,880 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:18,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,880 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:18,881 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:18,881 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:18,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:18,880 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,881 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,881 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,881 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,881 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:18,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:18,981 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:18,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:18,983 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:18,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:18,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:18,984 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:18,984 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:18,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:18,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,985 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831838987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,988 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:18,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831838987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,990 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,990 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:18,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:18,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:18,991 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,991 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:18,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:18,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:18,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:18,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:18,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,992 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:18,993 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:18,993 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:18,994 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,994 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:18,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:18,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d613b00[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:18,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:18,996 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,998 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:18,998 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:18,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:18,998 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:19,001 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:19,003 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:19,003 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:19,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:19,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:19,004 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:19,004 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:19,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,006 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:19,006 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,007 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,007 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:19,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:19,009 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:19,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:19,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:19,012 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:19,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:19,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:19,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:19,012 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:19,012 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:19,011 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:19,012 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:19,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 311 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831839066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,090 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=905 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/fc492764d13843fa915b8a57757a4157 2023-03-26 11:56:19,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/a822dab9cc894e85ae8e433a163d2ebe as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a822dab9cc894e85ae8e433a163d2ebe 2023-03-26 11:56:19,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a822dab9cc894e85ae8e433a163d2ebe, entries=400, sequenceid=905, filesize=59.8 K 2023-03-26 11:56:19,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/f75df679ddc54ef1b4a3ff05d28a9cc1 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/f75df679ddc54ef1b4a3ff05d28a9cc1 2023-03-26 11:56:19,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/f75df679ddc54ef1b4a3ff05d28a9cc1, entries=150, sequenceid=905, filesize=11.9 K 2023-03-26 11:56:19,103 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/fc492764d13843fa915b8a57757a4157 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fc492764d13843fa915b8a57757a4157 2023-03-26 11:56:19,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fc492764d13843fa915b8a57757a4157, entries=150, sequenceid=905, filesize=11.9 K 2023-03-26 11:56:19,106 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 5890b64a473c744d0cc3b74eead11b50 in 459ms, sequenceid=905, compaction requested=true 2023-03-26 11:56:19,106 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:19,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:19,106 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:19,106 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:19,106 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:19,107 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:19,108 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 140482 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:19,108 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:19,108 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:19,108 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ef788bf688430c9b8736bf5a0e3431, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/af37598183504812aca0495a713c7a4e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/36f14378cbfc4d68b7cd267c9fb8f9bb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a822dab9cc894e85ae8e433a163d2ebe] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=137.2 K 2023-03-26 11:56:19,108 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 97ef788bf688430c9b8736bf5a0e3431, keycount=150, bloomtype=ROW, size=26.5 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1679831775206 2023-03-26 11:56:19,108 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49834 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:19,109 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:19,109 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:19,109 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/05b2d1ace6f84a64a42be522a22ad206, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e29e388e8f9b41678e051ecd19ce829c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/041f0f64ff5e4aa7989f2e62b083f0c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/f75df679ddc54ef1b4a3ff05d28a9cc1] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=48.7 K 2023-03-26 11:56:19,109 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 05b2d1ace6f84a64a42be522a22ad206, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1679831775206 2023-03-26 11:56:19,109 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting af37598183504812aca0495a713c7a4e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=881, earliestPutTs=1679831775912 2023-03-26 11:56:19,109 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting e29e388e8f9b41678e051ecd19ce829c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=881, earliestPutTs=1679831775912 2023-03-26 11:56:19,110 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 36f14378cbfc4d68b7cd267c9fb8f9bb, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=894, earliestPutTs=1679831778002 2023-03-26 11:56:19,110 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 041f0f64ff5e4aa7989f2e62b083f0c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=894, earliestPutTs=1679831778002 2023-03-26 11:56:19,110 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting a822dab9cc894e85ae8e433a163d2ebe, keycount=400, bloomtype=ROW, size=59.8 K, encoding=NONE, compression=NONE, seqNum=905, earliestPutTs=1679831778634 2023-03-26 11:56:19,110 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f75df679ddc54ef1b4a3ff05d28a9cc1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=905, earliestPutTs=1679831778636 2023-03-26 11:56:19,112 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:19,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:19,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:19,115 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:19,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:19,116 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:19,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:19,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:19,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:19,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,116 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:19,118 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,118 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:19,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:19,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:19,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,119 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:19,119 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#604 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:19,119 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:19,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:19,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:19,120 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:19,120 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:19,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:19,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:19,120 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:19,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:19,121 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:19,121 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:19,121 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-03-26 11:56:19,123 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#603 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:19,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:19,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:19,308 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831839303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,309 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831839304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,374 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831839371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831839409, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831839410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,555 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/924f7fb8a5cc4118ac43a4c02a97ac44 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/924f7fb8a5cc4118ac43a4c02a97ac44 2023-03-26 11:56:19,560 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 924f7fb8a5cc4118ac43a4c02a97ac44(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:19,560 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:19,560 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=12, startTime=1679831779106; duration=0sec 2023-03-26 11:56:19,560 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:19,560 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:19,562 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:19,562 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:19,562 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:19,562 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8861e1bbe347413c8b47252a41f15a2e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/6d72fc7536d84ff1b6b43c3e7b54dfb9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c90b4da55e024e70a10681db77c74f28, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fc492764d13843fa915b8a57757a4157] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=48.6 K 2023-03-26 11:56:19,562 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303263645417c2890419baaca01fce66fa347 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263645417c2890419baaca01fce66fa347 2023-03-26 11:56:19,563 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 8861e1bbe347413c8b47252a41f15a2e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=844, earliestPutTs=1679831775206 2023-03-26 11:56:19,563 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 6d72fc7536d84ff1b6b43c3e7b54dfb9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=881, earliestPutTs=1679831775912 2023-03-26 11:56:19,563 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c90b4da55e024e70a10681db77c74f28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=894, earliestPutTs=1679831778002 2023-03-26 11:56:19,564 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fc492764d13843fa915b8a57757a4157, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=905, earliestPutTs=1679831778636 2023-03-26 11:56:19,568 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=941, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/7eeb575d612243028830b25b2fab311a 2023-03-26 11:56:19,571 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#606 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:19,578 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/fa8604659a624cfeb6ff636eb9433a01 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fa8604659a624cfeb6ff636eb9433a01 2023-03-26 11:56:19,587 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into fa8604659a624cfeb6ff636eb9433a01(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:19,587 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:19,587 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=12, startTime=1679831779107; duration=0sec 2023-03-26 11:56:19,587 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:19,617 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,617 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831839614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831839613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,877 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831839875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,923 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,923 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:19,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831839920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831839920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:19,960 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/4bfcc4baccb84fab8455b033de17da5d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4bfcc4baccb84fab8455b033de17da5d 2023-03-26 11:56:19,964 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 4bfcc4baccb84fab8455b033de17da5d(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:19,964 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:19,964 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=12, startTime=1679831779106; duration=0sec 2023-03-26 11:56:19,964 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:19,982 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=941 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/de6b40825b094e8e8bfd0b91f7ae4846 2023-03-26 11:56:19,991 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=941 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/3070c7e101fe4933b4a9fa88b5fa1c9b 2023-03-26 11:56:19,995 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/7eeb575d612243028830b25b2fab311a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7eeb575d612243028830b25b2fab311a 2023-03-26 11:56:19,998 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7eeb575d612243028830b25b2fab311a, entries=150, sequenceid=941, filesize=25.4 K 2023-03-26 11:56:19,999 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/de6b40825b094e8e8bfd0b91f7ae4846 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de6b40825b094e8e8bfd0b91f7ae4846 2023-03-26 11:56:20,002 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de6b40825b094e8e8bfd0b91f7ae4846, entries=150, sequenceid=941, filesize=11.9 K 2023-03-26 11:56:20,003 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/3070c7e101fe4933b4a9fa88b5fa1c9b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3070c7e101fe4933b4a9fa88b5fa1c9b 2023-03-26 11:56:20,006 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3070c7e101fe4933b4a9fa88b5fa1c9b, entries=150, sequenceid=941, filesize=11.9 K 2023-03-26 11:56:20,007 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 5890b64a473c744d0cc3b74eead11b50 in 886ms, sequenceid=941, compaction requested=false 2023-03-26 11:56:20,007 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:20,007 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:20,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:20,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:20,007 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:20,010 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,010 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:20,011 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3137e120[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:20,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:20,011 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,013 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:20,013 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:20,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,013 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:20,015 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,015 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,016 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:20,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:20,019 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:20,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:20,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:20,019 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:20,019 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:20,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,023 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,024 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,026 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,029 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,029 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:20,030 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:20,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,030 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:20,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,030 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:20,030 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:20,060 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-03-26 11:56:20,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:20,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:20,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,060 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:20,060 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,072 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326996cce569dcd4a5db9f0036f6353c1f2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326996cce569dcd4a5db9f0036f6353c1f2 2023-03-26 11:56:20,076 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=956, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/97ea73abb7904aa397e85c666da2729a 2023-03-26 11:56:20,085 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/0b8a3b647b1a480a9457f5889a396529 2023-03-26 11:56:20,094 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/bd18be34d9ad44bb8eddcf9e4f2b509d 2023-03-26 11:56:20,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/97ea73abb7904aa397e85c666da2729a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ea73abb7904aa397e85c666da2729a 2023-03-26 11:56:20,101 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ea73abb7904aa397e85c666da2729a, entries=150, sequenceid=956, filesize=25.4 K 2023-03-26 11:56:20,102 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/0b8a3b647b1a480a9457f5889a396529 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/0b8a3b647b1a480a9457f5889a396529 2023-03-26 11:56:20,105 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/0b8a3b647b1a480a9457f5889a396529, entries=150, sequenceid=956, filesize=11.9 K 2023-03-26 11:56:20,105 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/bd18be34d9ad44bb8eddcf9e4f2b509d as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd18be34d9ad44bb8eddcf9e4f2b509d 2023-03-26 11:56:20,108 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd18be34d9ad44bb8eddcf9e4f2b509d, entries=150, sequenceid=956, filesize=11.9 K 2023-03-26 11:56:20,109 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=87.22 KB/89310 for 5890b64a473c744d0cc3b74eead11b50 in 49ms, sequenceid=956, compaction requested=true 2023-03-26 11:56:20,109 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:20,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:20,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:20,109 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:20,109 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:20,109 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:20,110 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37784 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:20,110 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79364 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:20,110 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:20,110 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:20,110 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,110 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,110 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/924f7fb8a5cc4118ac43a4c02a97ac44, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de6b40825b094e8e8bfd0b91f7ae4846, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/0b8a3b647b1a480a9457f5889a396529] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=36.9 K 2023-03-26 11:56:20,110 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4bfcc4baccb84fab8455b033de17da5d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7eeb575d612243028830b25b2fab311a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ea73abb7904aa397e85c666da2729a] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=77.5 K 2023-03-26 11:56:20,111 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 924f7fb8a5cc4118ac43a4c02a97ac44, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=905, earliestPutTs=1679831778636 2023-03-26 11:56:20,111 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting de6b40825b094e8e8bfd0b91f7ae4846, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679831778738 2023-03-26 11:56:20,111 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 4bfcc4baccb84fab8455b033de17da5d, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=905, earliestPutTs=1679831778636 2023-03-26 11:56:20,111 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 0b8a3b647b1a480a9457f5889a396529, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1679831779302 2023-03-26 11:56:20,112 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 7eeb575d612243028830b25b2fab311a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679831778738 2023-03-26 11:56:20,112 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 97ea73abb7904aa397e85c666da2729a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1679831779302 2023-03-26 11:56:20,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:20,120 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-03-26 11:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,121 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#612 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:20,121 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,129 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#613 average throughput is 2.19 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:20,130 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:20,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:20,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:20,137 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/06c815cba1e8403496990abd27a71651 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/06c815cba1e8403496990abd27a71651 2023-03-26 11:56:20,141 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 06c815cba1e8403496990abd27a71651(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:20,141 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:20,141 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=13, startTime=1679831780109; duration=0sec 2023-03-26 11:56:20,142 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:20,142 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-03-26 11:56:20,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:20,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:20,143 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:20,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:20,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:20,143 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37716 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-03-26 11:56:20,143 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:20,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,143 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,143 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,143 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fa8604659a624cfeb6ff636eb9433a01, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3070c7e101fe4933b4a9fa88b5fa1c9b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd18be34d9ad44bb8eddcf9e4f2b509d] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=36.8 K 2023-03-26 11:56:20,144 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting fa8604659a624cfeb6ff636eb9433a01, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=905, earliestPutTs=1679831778636 2023-03-26 11:56:20,144 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 3070c7e101fe4933b4a9fa88b5fa1c9b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=941, earliestPutTs=1679831778738 2023-03-26 11:56:20,145 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,145 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bd18be34d9ad44bb8eddcf9e4f2b509d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1679831779302 2023-03-26 11:56:20,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,145 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:20,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:20,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:20,147 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:20,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:20,147 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303266b972cb3e4b04bacb708392483985bd9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303266b972cb3e4b04bacb708392483985bd9 2023-03-26 11:56:20,147 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,147 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:20,147 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:20,147 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:20,147 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:20,148 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:20,150 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,150 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,151 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:20,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a21dd72[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:20,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:20,152 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,154 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:20,154 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:20,154 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,154 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:20,155 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:20,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,155 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,156 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,158 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#615 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:20,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:20,159 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:20,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:20,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:20,159 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:20,159 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:20,161 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,161 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,161 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,162 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,163 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,164 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=973, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/b1ac0a24e5c0443ea1261ff3e39320de 2023-03-26 11:56:20,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,167 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,167 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,167 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:20,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:20,166 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,167 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:20,167 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:20,167 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:20,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,169 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831840165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,176 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/bc5810cb4e524d3a9988c8fde4fd57f2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/bc5810cb4e524d3a9988c8fde4fd57f2 2023-03-26 11:56:20,181 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into bc5810cb4e524d3a9988c8fde4fd57f2(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:20,181 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:20,181 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c11cee48fe63460f976080fc23efc441 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c11cee48fe63460f976080fc23efc441 2023-03-26 11:56:20,181 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=13, startTime=1679831780109; duration=0sec 2023-03-26 11:56:20,181 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:20,188 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=973 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/5863e99bb17e423c8c97ca20cc742366 2023-03-26 11:56:20,190 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into c11cee48fe63460f976080fc23efc441(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:20,190 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:20,190 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=13, startTime=1679831780109; duration=0sec 2023-03-26 11:56:20,190 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:20,267 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:20,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:20,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:20,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:20,271 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:20,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:20,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:20,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,271 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,272 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831840270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,272 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,272 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,273 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:20,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,274 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:20,274 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:20,274 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:20,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:20,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:20,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:20,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:20,274 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,274 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:20,274 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:20,274 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,274 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:20,275 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:20,276 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,276 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,277 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:20,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4cfc485e[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:20,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:20,278 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,280 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:20,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,280 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:20,280 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:20,283 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,283 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:20,285 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:20,285 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:20,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:20,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:20,285 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:20,285 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:20,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,287 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,288 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,289 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:20,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,292 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,292 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:20,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:20,293 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:20,293 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:20,393 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:20,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:20,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:20,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:20,396 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:20,396 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:20,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:20,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:20,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,396 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,398 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,398 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,399 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:20,399 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:20,399 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:20,399 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,399 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:20,399 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:20,399 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:20,399 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:20,401 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,402 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:20,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f1d3fa[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:20,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:20,403 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,404 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:20,404 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:20,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,405 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:20,406 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:20,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,406 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:20,408 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:20,408 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:20,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:20,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:20,408 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:20,408 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:20,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,410 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,410 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,411 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,414 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:20,415 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,415 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,415 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,415 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:20,415 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:20,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,427 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831840426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,430 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 280 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831840428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 334 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831840474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,515 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:20,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:20,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:20,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:20,519 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:20,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:20,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:20,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,519 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,522 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,522 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:20,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,523 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:20,523 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:20,523 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:20,523 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:20,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:20,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:20,524 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:20,524 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:20,526 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,526 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:20,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@76410325[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:20,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:20,528 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,529 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:20,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:20,530 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,530 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:20,530 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:20,532 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,532 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:20,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:20,534 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:20,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:20,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:20,534 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:20,534 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:20,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,536 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:20,537 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:20,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,538 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:20,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:20,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:20,541 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,541 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:20,541 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:20,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:20,541 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,541 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:20,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:20,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:20,597 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=973 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/bbfcac663d7e4f2a8c200a87c2c6d5ec 2023-03-26 11:56:20,601 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/b1ac0a24e5c0443ea1261ff3e39320de as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b1ac0a24e5c0443ea1261ff3e39320de 2023-03-26 11:56:20,604 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b1ac0a24e5c0443ea1261ff3e39320de, entries=200, sequenceid=973, filesize=32.3 K 2023-03-26 11:56:20,605 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/5863e99bb17e423c8c97ca20cc742366 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5863e99bb17e423c8c97ca20cc742366 2023-03-26 11:56:20,608 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5863e99bb17e423c8c97ca20cc742366, entries=150, sequenceid=973, filesize=11.9 K 2023-03-26 11:56:20,609 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/bbfcac663d7e4f2a8c200a87c2c6d5ec as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bbfcac663d7e4f2a8c200a87c2c6d5ec 2023-03-26 11:56:20,612 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bbfcac663d7e4f2a8c200a87c2c6d5ec, entries=150, sequenceid=973, filesize=11.9 K 2023-03-26 11:56:20,612 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 5890b64a473c744d0cc3b74eead11b50 in 492ms, sequenceid=973, compaction requested=false 2023-03-26 11:56:20,613 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:20,641 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:20,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:20,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:20,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:20,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:20,645 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:20,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:20,645 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:20,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,646 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,651 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:20,651 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:20,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,651 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,651 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:20,651 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:20,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:20,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:20,652 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:20,653 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:20,653 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:20,672 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a8b1711494b64f9e94ef06fedde93f0a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a8b1711494b64f9e94ef06fedde93f0a 2023-03-26 11:56:20,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:20,777 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:20,816 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 350 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831840815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831840882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:20,919 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:20,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=38971] ipc.CallRunner(144): callId: 352 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831840917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,081 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=991, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/ba0c3b729e154927874658b71d105fe4 2023-03-26 11:56:21,122 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:21,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 354 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831841121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,426 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:21,427 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 356 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831841425, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,434 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:21,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831841432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,437 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:21,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 282 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831841437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,490 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=991 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/d628762ef3234028aab4b677deb5ae1a 2023-03-26 11:56:21,901 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=991 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/f5624f8b58a742e3a9c11c6e82429ccd 2023-03-26 11:56:21,905 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/ba0c3b729e154927874658b71d105fe4 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ba0c3b729e154927874658b71d105fe4 2023-03-26 11:56:21,908 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ba0c3b729e154927874658b71d105fe4, entries=150, sequenceid=991, filesize=25.4 K 2023-03-26 11:56:21,909 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/d628762ef3234028aab4b677deb5ae1a as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/d628762ef3234028aab4b677deb5ae1a 2023-03-26 11:56:21,912 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/d628762ef3234028aab4b677deb5ae1a, entries=150, sequenceid=991, filesize=11.9 K 2023-03-26 11:56:21,912 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/f5624f8b58a742e3a9c11c6e82429ccd as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f5624f8b58a742e3a9c11c6e82429ccd 2023-03-26 11:56:21,915 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f5624f8b58a742e3a9c11c6e82429ccd, entries=150, sequenceid=991, filesize=11.9 K 2023-03-26 11:56:21,916 INFO [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 5890b64a473c744d0cc3b74eead11b50 in 1263ms, sequenceid=991, compaction requested=true 2023-03-26 11:56:21,916 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:21,916 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:21,916 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:21,918 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,918 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:21,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:21,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ff9ad2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:21,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:21,919 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,921 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:21,921 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:21,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,921 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:21,923 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:21,923 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:21,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:21,926 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:21,926 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:21,926 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:21,926 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:21,926 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:21,926 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:21,929 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,929 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,929 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:21,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,930 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:21,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:21,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:21,932 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-03-26 11:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:21,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,932 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,932 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:21,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:21,936 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:21,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:21,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:21,936 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:21,936 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:21,936 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:21,936 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:21,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:21,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:21,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:21,964 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831841964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,036 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,038 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:22,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:22,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:22,040 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,040 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,041 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,041 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,042 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:22,043 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:22,043 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:22,043 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,043 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:22,043 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:22,043 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,043 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:22,044 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:22,045 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,045 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:22,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c6f8c6b[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:22,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:22,047 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,053 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,053 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:22,053 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:22,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,053 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:22,055 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,055 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,055 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,055 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:22,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:22,057 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:22,057 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:22,057 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:22,057 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:22,057 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:22,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,059 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,059 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,061 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:22,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:22,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,063 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,063 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,064 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:22,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,063 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:22,064 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:22,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:22,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831842065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,164 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:22,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:22,167 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:22,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:22,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:22,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,167 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,169 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,169 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,170 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,170 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:22,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:22,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,171 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:22,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:22,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,171 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:22,172 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,172 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:22,172 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:22,172 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:22,172 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:22,173 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,173 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,174 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:22,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5af7f41f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:22,175 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:22,175 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,176 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:22,177 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:22,177 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,177 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:22,178 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,178 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:22,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:22,180 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:22,180 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:22,180 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:22,180 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:22,180 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:22,180 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:22,183 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,183 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,183 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,183 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,183 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,184 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,187 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:22,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:22,187 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,188 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:22,188 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:22,188 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:22,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,188 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:22,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 374 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831842267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,288 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,290 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:22,291 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:22,291 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:22,291 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,291 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:22,291 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:22,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,292 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,293 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,293 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,293 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,294 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:22,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,294 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:22,295 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:22,295 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:22,295 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,295 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:22,295 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:22,295 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:22,295 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:22,296 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:22,297 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,297 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:22,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@24518c4f[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:22,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:22,300 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,304 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,305 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:22,305 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:22,305 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:22,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,305 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:22,306 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,306 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:22,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:22,309 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:22,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:22,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:22,309 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:22,309 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:22,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,311 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,311 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,313 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:22,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-03-26 11:56:22,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:22,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,315 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,315 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,315 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:22,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,316 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,316 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:22,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,346 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202303267ea03a8a551f4bd49ee8fae3c24d624e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267ea03a8a551f4bd49ee8fae3c24d624e 2023-03-26 11:56:22,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:22,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35794 deadline: 1679831842381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,384 DEBUG [Thread-3664] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8175 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:56:22,416 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:22,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:22,419 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:22,419 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,419 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:22,419 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:22,420 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,420 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,422 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,422 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,423 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:22,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:22,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:22,424 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:22,424 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,426 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:22,426 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:22,428 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a06bb1d[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:22,428 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:22,428 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,429 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,429 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,429 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:22,429 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:22,431 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,431 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:22,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:22,433 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:22,433 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:22,433 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:22,433 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:22,433 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:22,436 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,436 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,436 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,436 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,438 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,440 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,441 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,441 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:22,441 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:22,441 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,441 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:22,441 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:22,541 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,542 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,543 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:22,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:22,545 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:22,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:22,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:22,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,545 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,547 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,547 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,547 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,547 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,547 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:22,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,548 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:22,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,548 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:22,548 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:22,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,548 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:22,549 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,549 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:22,549 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:22,549 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:22,549 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:22,551 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,551 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,553 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:22,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ada78e2[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:22,554 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:22,554 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,556 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:22,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,556 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:22,556 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:22,557 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,557 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:22,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:22,560 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:22,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:22,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:22,560 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:22,560 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:22,562 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,562 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,562 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,563 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,565 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,567 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,567 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,567 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,567 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,567 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,567 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,567 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,567 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:22,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:22,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,568 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,568 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:22,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,568 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:22,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:22,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=38971] ipc.CallRunner(144): callId: 376 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831842571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,668 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-03-26 11:56:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-03-26 11:56:22,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-03-26 11:56:22,672 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-03-26 11:56:22,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-03-26 11:56:22,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-03-26 11:56:22,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,672 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,674 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,674 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-03-26 11:56:22,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-03-26 11:56:22,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-03-26 11:56:22,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-03-26 11:56:22,675 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-03-26 11:56:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-03-26 11:56:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-03-26 11:56:22,675 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-03-26 11:56:22,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. started... 2023-03-26 11:56:22,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. as already flushing 2023-03-26 11:56:22,676 DEBUG [rs(jenkins-hbase4.apache.org,38971,1679831615531)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-03-26 11:56:22,676 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-03-26 11:56:22,678 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,678 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-03-26 11:56:22,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,38971,1679831615531' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-03-26 11:56:22,679 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f6822aa[Count = 0] remaining members to acquire global barrier 2023-03-26 11:56:22,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-03-26 11:56:22,679 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,681 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-03-26 11:56:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-03-26 11:56:22,681 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,38971,1679831615531' in zk 2023-03-26 11:56:22,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,681 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-03-26 11:56:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-03-26 11:56:22,685 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,685 DEBUG [member: 'jenkins-hbase4.apache.org,38971,1679831615531' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-03-26 11:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,38971,1679831615531': 2023-03-26 11:56:22,688 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,38971,1679831615531' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-03-26 11:56:22,688 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-03-26 11:56:22,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-03-26 11:56:22,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-03-26 11:56:22,688 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-03-26 11:56:22,688 INFO [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-03-26 11:56:22,689 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,689 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-03-26 11:56:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-03-26 11:56:22,689 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,689 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-03-26 11:56:22,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,690 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-03-26 11:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,693 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-03-26 11:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-03-26 11:56:22,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-03-26 11:56:22,695 DEBUG [(jenkins-hbase4.apache.org,43599,1679831614279)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-03-26 11:56:22,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-03-26 11:56:22,695 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-03-26 11:56:22,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-03-26 11:56:22,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-03-26 11:56:22,695 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,696 DEBUG [Thread-3670] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-03-26 11:56:22,696 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-03-26 11:56:22,696 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-03-26 11:56:22,696 DEBUG [Thread-3670] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-03-26 11:56:22,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-03-26 11:56:22,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-03-26 11:56:22,723 DEBUG [Thread-3679] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3dca23c0 to 127.0.0.1:53955 2023-03-26 11:56:22,723 DEBUG [Thread-3679] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:22,724 DEBUG [Thread-3675] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0a470a3d to 127.0.0.1:53955 2023-03-26 11:56:22,724 DEBUG [Thread-3675] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:22,724 DEBUG [Thread-3671] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e65a76 to 127.0.0.1:53955 2023-03-26 11:56:22,725 DEBUG [Thread-3671] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:22,728 DEBUG [Thread-3673] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e6a6f3c to 127.0.0.1:53955 2023-03-26 11:56:22,728 DEBUG [Thread-3673] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:22,728 DEBUG [Thread-3677] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4bb72437 to 127.0.0.1:53955 2023-03-26 11:56:22,728 DEBUG [Thread-3677] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:22,761 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1009, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/85fbe16b5b804a7582d5aa932683d239 2023-03-26 11:56:22,796 DEBUG [Thread-3670] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-03-26 11:56:22,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-03-26 11:56:22,893 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:22,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35802 deadline: 1679831842893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:22,894 DEBUG [Thread-3666] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:56:23,076 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:23,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 378 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35824 deadline: 1679831843076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:23,169 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/b8062d0f7ecb4820929f83b203f45a12 2023-03-26 11:56:23,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:23,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 284 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35762 deadline: 1679831843449, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:23,451 DEBUG [Thread-3668] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:56:23,451 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-03-26 11:56:23,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:35780 deadline: 1679831843451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:23,453 DEBUG [Thread-3662] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., hostname=jenkins-hbase4.apache.org,38971,1679831615531, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor32.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-03-26 11:56:23,577 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=1009 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c93aeb9c53a940c5a2a566a5664093a8 2023-03-26 11:56:23,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/85fbe16b5b804a7582d5aa932683d239 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/85fbe16b5b804a7582d5aa932683d239 2023-03-26 11:56:23,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/85fbe16b5b804a7582d5aa932683d239, entries=200, sequenceid=1009, filesize=32.3 K 2023-03-26 11:56:23,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/b8062d0f7ecb4820929f83b203f45a12 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/b8062d0f7ecb4820929f83b203f45a12 2023-03-26 11:56:23,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/b8062d0f7ecb4820929f83b203f45a12, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-26 11:56:23,587 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/c93aeb9c53a940c5a2a566a5664093a8 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c93aeb9c53a940c5a2a566a5664093a8 2023-03-26 11:56:23,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c93aeb9c53a940c5a2a566a5664093a8, entries=150, sequenceid=1009, filesize=11.9 K 2023-03-26 11:56:23,590 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 5890b64a473c744d0cc3b74eead11b50 in 1658ms, sequenceid=1009, compaction requested=true 2023-03-26 11:56:23,590 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:23,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:23,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:23,590 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:23,591 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:23,590 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:23,591 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 119592 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:23,591 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50072 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:23,592 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/A is initiating minor compaction (all files) 2023-03-26 11:56:23,592 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/B is initiating minor compaction (all files) 2023-03-26 11:56:23,592 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/A in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:23,592 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/B in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:23,592 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/bc5810cb4e524d3a9988c8fde4fd57f2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b1ac0a24e5c0443ea1261ff3e39320de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ba0c3b729e154927874658b71d105fe4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/85fbe16b5b804a7582d5aa932683d239] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=116.8 K 2023-03-26 11:56:23,592 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/06c815cba1e8403496990abd27a71651, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5863e99bb17e423c8c97ca20cc742366, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/d628762ef3234028aab4b677deb5ae1a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/b8062d0f7ecb4820929f83b203f45a12] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=48.9 K 2023-03-26 11:56:23,592 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting bc5810cb4e524d3a9988c8fde4fd57f2, keycount=150, bloomtype=ROW, size=26.7 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1679831779302 2023-03-26 11:56:23,592 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 06c815cba1e8403496990abd27a71651, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1679831779302 2023-03-26 11:56:23,592 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting b1ac0a24e5c0443ea1261ff3e39320de, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=973, earliestPutTs=1679831780089 2023-03-26 11:56:23,592 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting 5863e99bb17e423c8c97ca20cc742366, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=973, earliestPutTs=1679831780092 2023-03-26 11:56:23,593 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting ba0c3b729e154927874658b71d105fe4, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679831780141 2023-03-26 11:56:23,593 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting d628762ef3234028aab4b677deb5ae1a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679831780141 2023-03-26 11:56:23,593 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] compactions.Compactor(207): Compacting 85fbe16b5b804a7582d5aa932683d239, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1679831780807 2023-03-26 11:56:23,593 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting b8062d0f7ecb4820929f83b203f45a12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1679831780807 2023-03-26 11:56:23,599 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#B#compaction#624 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:23,601 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#A#compaction#625 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:24,008 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/616afd87e6304238845d6e5c55986880 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/616afd87e6304238845d6e5c55986880 2023-03-26 11:56:24,012 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/B of 5890b64a473c744d0cc3b74eead11b50 into 616afd87e6304238845d6e5c55986880(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:24,012 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:24,012 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/B, priority=12, startTime=1679831783590; duration=0sec 2023-03-26 11:56:24,013 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-03-26 11:56:24,013 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-03-26 11:56:24,014 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50004 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-03-26 11:56:24,014 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1912): 5890b64a473c744d0cc3b74eead11b50/C is initiating minor compaction (all files) 2023-03-26 11:56:24,014 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 5890b64a473c744d0cc3b74eead11b50/C in TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:24,014 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c11cee48fe63460f976080fc23efc441, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bbfcac663d7e4f2a8c200a87c2c6d5ec, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f5624f8b58a742e3a9c11c6e82429ccd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c93aeb9c53a940c5a2a566a5664093a8] into tmpdir=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp, totalSize=48.8 K 2023-03-26 11:56:24,014 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c11cee48fe63460f976080fc23efc441, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1679831779302 2023-03-26 11:56:24,014 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting bbfcac663d7e4f2a8c200a87c2c6d5ec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=973, earliestPutTs=1679831780092 2023-03-26 11:56:24,015 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting f5624f8b58a742e3a9c11c6e82429ccd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=991, earliestPutTs=1679831780141 2023-03-26 11:56:24,015 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] compactions.Compactor(207): Compacting c93aeb9c53a940c5a2a566a5664093a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=1009, earliestPutTs=1679831780807 2023-03-26 11:56:24,020 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] throttle.PressureAwareThroughputController(145): 5890b64a473c744d0cc3b74eead11b50#C#compaction#626 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-03-26 11:56:24,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=38971] regionserver.HRegion(9158): Flush requested on 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:24,081 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-03-26 11:56:24,081 DEBUG [Thread-3660] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63da10ca to 127.0.0.1:53955 2023-03-26 11:56:24,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:24,081 DEBUG [Thread-3660] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:24,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:24,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:24,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:24,081 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:24,081 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:24,415 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/4cf6e028123d47a886594d57f361bd49 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4cf6e028123d47a886594d57f361bd49 2023-03-26 11:56:24,418 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/A of 5890b64a473c744d0cc3b74eead11b50 into 4cf6e028123d47a886594d57f361bd49(size=26.9 K), total size for store is 26.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:24,419 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:24,419 INFO [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/A, priority=12, startTime=1679831783590; duration=0sec 2023-03-26 11:56:24,419 DEBUG [RS:0;jenkins-hbase4:38971-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:24,426 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/58d03033a3194eee978ee3ca171d406f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/58d03033a3194eee978ee3ca171d406f 2023-03-26 11:56:24,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 5890b64a473c744d0cc3b74eead11b50/C of 5890b64a473c744d0cc3b74eead11b50 into 58d03033a3194eee978ee3ca171d406f(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-03-26 11:56:24,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:24,429 INFO [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50., storeName=5890b64a473c744d0cc3b74eead11b50/C, priority=12, startTime=1679831783590; duration=0sec 2023-03-26 11:56:24,429 DEBUG [RS:0;jenkins-hbase4:38971-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-03-26 11:56:24,492 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023032682af3dd08c744eb489bdb9c9b9f2801b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682af3dd08c744eb489bdb9c9b9f2801b 2023-03-26 11:56:24,896 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1025, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/ad994567153a433c8dfbd88a401d7525 2023-03-26 11:56:25,304 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/a284538616d9475b94c971a83fe57a2b 2023-03-26 11:56:25,712 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=1025 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/f0ae9b27c71e4b6daef09dc8ccfeab5f 2023-03-26 11:56:25,715 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/ad994567153a433c8dfbd88a401d7525 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ad994567153a433c8dfbd88a401d7525 2023-03-26 11:56:25,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ad994567153a433c8dfbd88a401d7525, entries=150, sequenceid=1025, filesize=25.4 K 2023-03-26 11:56:25,719 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/a284538616d9475b94c971a83fe57a2b as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a284538616d9475b94c971a83fe57a2b 2023-03-26 11:56:25,722 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a284538616d9475b94c971a83fe57a2b, entries=150, sequenceid=1025, filesize=11.9 K 2023-03-26 11:56:25,722 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/f0ae9b27c71e4b6daef09dc8ccfeab5f as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f0ae9b27c71e4b6daef09dc8ccfeab5f 2023-03-26 11:56:25,725 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f0ae9b27c71e4b6daef09dc8ccfeab5f, entries=150, sequenceid=1025, filesize=11.9 K 2023-03-26 11:56:25,726 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 5890b64a473c744d0cc3b74eead11b50 in 1645ms, sequenceid=1025, compaction requested=false 2023-03-26 11:56:25,726 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:26,911 DEBUG [Thread-3666] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66b5b1f3 to 127.0.0.1:53955 2023-03-26 11:56:26,911 DEBUG [Thread-3666] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:27,476 DEBUG [Thread-3662] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59185ba7 to 127.0.0.1:53955 2023-03-26 11:56:27,476 DEBUG [Thread-3662] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:27,483 DEBUG [Thread-3668] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x544a4215 to 127.0.0.1:53955 2023-03-26 11:56:27,483 DEBUG [Thread-3668] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:32,399 DEBUG [Thread-3664] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x062b3caa to 127.0.0.1:53955 2023-03-26 11:56:32,400 DEBUG [Thread-3664] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 229 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 206 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 128 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 188 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(394): wrote 143 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(396): Readers: 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4496 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4550 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4513 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4494 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(398): read 4521 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-03-26 11:56:32,400 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:56:32,400 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x03ced4ba to 127.0.0.1:53955 2023-03-26 11:56:32,400 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:32,401 INFO [Listener at localhost/46409] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-03-26 11:56:32,401 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-03-26 11:56:32,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:32,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 11:56:32,404 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831792404"}]},"ts":"1679831792404"} 2023-03-26 11:56:32,405 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-03-26 11:56:32,406 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-03-26 11:56:32,407 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, UNASSIGN}] 2023-03-26 11:56:32,408 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, UNASSIGN 2023-03-26 11:56:32,409 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:32,409 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831792409"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1679831792409"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1679831792409"}]},"ts":"1679831792409"} 2023-03-26 11:56:32,410 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531}] 2023-03-26 11:56:32,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 11:56:32,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:32,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 5890b64a473c744d0cc3b74eead11b50, disabling compactions & flushes 2023-03-26 11:56:32,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:32,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:32,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. after waiting 0 ms 2023-03-26 11:56:32,561 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:32,561 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 5890b64a473c744d0cc3b74eead11b50 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-03-26 11:56:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=A 2023-03-26 11:56:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=B 2023-03-26 11:56:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 5890b64a473c744d0cc3b74eead11b50, store=C 2023-03-26 11:56:32,562 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-03-26 11:56:32,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 11:56:32,972 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230326a985c8f2beda4e3289d1b23dfe7608ca to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a985c8f2beda4e3289d1b23dfe7608ca 2023-03-26 11:56:33,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 11:56:33,377 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1034, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/8c197e4601d84096b3090ade67bfe97c 2023-03-26 11:56:33,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 11:56:33,784 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/22c523b8f1eb455ea860074fedd2e9cf 2023-03-26 11:56:33,792 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=1034 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/9a028c87b7d94e1c924bf9353aa380de 2023-03-26 11:56:33,794 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/A/8c197e4601d84096b3090ade67bfe97c as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8c197e4601d84096b3090ade67bfe97c 2023-03-26 11:56:33,797 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8c197e4601d84096b3090ade67bfe97c, entries=150, sequenceid=1034, filesize=25.4 K 2023-03-26 11:56:33,797 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/B/22c523b8f1eb455ea860074fedd2e9cf as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/22c523b8f1eb455ea860074fedd2e9cf 2023-03-26 11:56:33,800 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/22c523b8f1eb455ea860074fedd2e9cf, entries=150, sequenceid=1034, filesize=11.9 K 2023-03-26 11:56:33,801 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/.tmp/C/9a028c87b7d94e1c924bf9353aa380de as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9a028c87b7d94e1c924bf9353aa380de 2023-03-26 11:56:33,803 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9a028c87b7d94e1c924bf9353aa380de, entries=150, sequenceid=1034, filesize=11.9 K 2023-03-26 11:56:33,804 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 5890b64a473c744d0cc3b74eead11b50 in 1243ms, sequenceid=1034, compaction requested=true 2023-03-26 11:56:33,835 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ef0e16e0673345a697c026fb8f1d3663, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/76d02416f6bf41c9a8e25214b63c83b2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/980c0e8978314f15a8c0914de0ed3adc, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/269d925e977f4a60b5733de65e995713, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b7e40145d554ce89b2443699c2d26c6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/da0b1a3a95fb4b40bd8f9b032dd2798d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a66104a2c84e4700bbc346ca5e2c17b5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96ff62b902f34b3dae9155e469aba053, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4aa09191b44d4d48ab95d7842f476173, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8f50bc98398d45f7be237f2379ab2437, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/160784b63eb94d168df85140b7bbdd3f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0e18c695bf044b4495af2374a2a17005, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/aceac7e737964727b66a59f9d3a348f9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7e4ffe09120d450bad852bc61e9a5444, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/04f0aa89a298451db9f3051b5db51959, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/6ab445453111461db271a11c54fb5078, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b640976a5ef47528c43f42b23f398cb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8159cc6a8bd4436db877ddc3896b338a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b72fa7503e0049f6a58e77fd7899f59d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b05b4b46bdea42369b1c581f7dd07df8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/cf965ca273d44e0d902a914ce99c2be8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0cbc503d1a6c4c38b42327c2313f857e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c113121e841e44e89885641e37b9f7a5, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96365ea8f4564857977a0ef1e165a076, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/24b8b494506d4bfe9c27347ac87a06da, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a130bf9b6796476b8e893ddb9c0b1448, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c9b3af229bf147e98a8e7975deaa1ace, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/fa26320c60c841039bddee0a1abe668d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/79f3fefa97d944e980a28860e168a733, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3def535288cf41a1a2a3be467933dfd9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/e5ae6a7d5e254b1784488e125961bfa2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ef788bf688430c9b8736bf5a0e3431, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/af37598183504812aca0495a713c7a4e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/36f14378cbfc4d68b7cd267c9fb8f9bb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a822dab9cc894e85ae8e433a163d2ebe, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4bfcc4baccb84fab8455b033de17da5d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7eeb575d612243028830b25b2fab311a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/bc5810cb4e524d3a9988c8fde4fd57f2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ea73abb7904aa397e85c666da2729a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b1ac0a24e5c0443ea1261ff3e39320de, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ba0c3b729e154927874658b71d105fe4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/85fbe16b5b804a7582d5aa932683d239] to archive 2023-03-26 11:56:33,836 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:56:33,837 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ef0e16e0673345a697c026fb8f1d3663 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ef0e16e0673345a697c026fb8f1d3663 2023-03-26 11:56:33,838 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/76d02416f6bf41c9a8e25214b63c83b2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/76d02416f6bf41c9a8e25214b63c83b2 2023-03-26 11:56:33,839 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/980c0e8978314f15a8c0914de0ed3adc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/980c0e8978314f15a8c0914de0ed3adc 2023-03-26 11:56:33,840 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/269d925e977f4a60b5733de65e995713 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/269d925e977f4a60b5733de65e995713 2023-03-26 11:56:33,841 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b7e40145d554ce89b2443699c2d26c6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b7e40145d554ce89b2443699c2d26c6 2023-03-26 11:56:33,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/da0b1a3a95fb4b40bd8f9b032dd2798d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/da0b1a3a95fb4b40bd8f9b032dd2798d 2023-03-26 11:56:33,842 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a66104a2c84e4700bbc346ca5e2c17b5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a66104a2c84e4700bbc346ca5e2c17b5 2023-03-26 11:56:33,843 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96ff62b902f34b3dae9155e469aba053 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96ff62b902f34b3dae9155e469aba053 2023-03-26 11:56:33,844 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4aa09191b44d4d48ab95d7842f476173 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4aa09191b44d4d48ab95d7842f476173 2023-03-26 11:56:33,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8f50bc98398d45f7be237f2379ab2437 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8f50bc98398d45f7be237f2379ab2437 2023-03-26 11:56:33,845 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/160784b63eb94d168df85140b7bbdd3f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/160784b63eb94d168df85140b7bbdd3f 2023-03-26 11:56:33,846 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0e18c695bf044b4495af2374a2a17005 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0e18c695bf044b4495af2374a2a17005 2023-03-26 11:56:33,847 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/aceac7e737964727b66a59f9d3a348f9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/aceac7e737964727b66a59f9d3a348f9 2023-03-26 11:56:33,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7e4ffe09120d450bad852bc61e9a5444 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7e4ffe09120d450bad852bc61e9a5444 2023-03-26 11:56:33,848 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/f8e6dbdf6b0f4f5ca9acaf8f28ecc8ec 2023-03-26 11:56:33,849 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/04f0aa89a298451db9f3051b5db51959 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/04f0aa89a298451db9f3051b5db51959 2023-03-26 11:56:33,850 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/6ab445453111461db271a11c54fb5078 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/6ab445453111461db271a11c54fb5078 2023-03-26 11:56:33,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b640976a5ef47528c43f42b23f398cb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3b640976a5ef47528c43f42b23f398cb 2023-03-26 11:56:33,851 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8159cc6a8bd4436db877ddc3896b338a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8159cc6a8bd4436db877ddc3896b338a 2023-03-26 11:56:33,852 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b72fa7503e0049f6a58e77fd7899f59d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b72fa7503e0049f6a58e77fd7899f59d 2023-03-26 11:56:33,853 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b05b4b46bdea42369b1c581f7dd07df8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b05b4b46bdea42369b1c581f7dd07df8 2023-03-26 11:56:33,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/cf965ca273d44e0d902a914ce99c2be8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/cf965ca273d44e0d902a914ce99c2be8 2023-03-26 11:56:33,854 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0cbc503d1a6c4c38b42327c2313f857e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/0cbc503d1a6c4c38b42327c2313f857e 2023-03-26 11:56:33,855 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c113121e841e44e89885641e37b9f7a5 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c113121e841e44e89885641e37b9f7a5 2023-03-26 11:56:33,856 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96365ea8f4564857977a0ef1e165a076 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/96365ea8f4564857977a0ef1e165a076 2023-03-26 11:56:33,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/24b8b494506d4bfe9c27347ac87a06da to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/24b8b494506d4bfe9c27347ac87a06da 2023-03-26 11:56:33,857 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a130bf9b6796476b8e893ddb9c0b1448 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a130bf9b6796476b8e893ddb9c0b1448 2023-03-26 11:56:33,858 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c9b3af229bf147e98a8e7975deaa1ace to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/c9b3af229bf147e98a8e7975deaa1ace 2023-03-26 11:56:33,859 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/fa26320c60c841039bddee0a1abe668d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/fa26320c60c841039bddee0a1abe668d 2023-03-26 11:56:33,860 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/79f3fefa97d944e980a28860e168a733 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/79f3fefa97d944e980a28860e168a733 2023-03-26 11:56:33,860 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3def535288cf41a1a2a3be467933dfd9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/3def535288cf41a1a2a3be467933dfd9 2023-03-26 11:56:33,861 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/e5ae6a7d5e254b1784488e125961bfa2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/e5ae6a7d5e254b1784488e125961bfa2 2023-03-26 11:56:33,862 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ef788bf688430c9b8736bf5a0e3431 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ef788bf688430c9b8736bf5a0e3431 2023-03-26 11:56:33,862 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/af37598183504812aca0495a713c7a4e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/af37598183504812aca0495a713c7a4e 2023-03-26 11:56:33,863 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/36f14378cbfc4d68b7cd267c9fb8f9bb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/36f14378cbfc4d68b7cd267c9fb8f9bb 2023-03-26 11:56:33,864 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a822dab9cc894e85ae8e433a163d2ebe to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/a822dab9cc894e85ae8e433a163d2ebe 2023-03-26 11:56:33,865 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4bfcc4baccb84fab8455b033de17da5d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4bfcc4baccb84fab8455b033de17da5d 2023-03-26 11:56:33,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7eeb575d612243028830b25b2fab311a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/7eeb575d612243028830b25b2fab311a 2023-03-26 11:56:33,866 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/bc5810cb4e524d3a9988c8fde4fd57f2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/bc5810cb4e524d3a9988c8fde4fd57f2 2023-03-26 11:56:33,867 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ea73abb7904aa397e85c666da2729a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/97ea73abb7904aa397e85c666da2729a 2023-03-26 11:56:33,868 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b1ac0a24e5c0443ea1261ff3e39320de to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/b1ac0a24e5c0443ea1261ff3e39320de 2023-03-26 11:56:33,868 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ba0c3b729e154927874658b71d105fe4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ba0c3b729e154927874658b71d105fe4 2023-03-26 11:56:33,869 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/85fbe16b5b804a7582d5aa932683d239 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/85fbe16b5b804a7582d5aa932683d239 2023-03-26 11:56:33,872 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3d526f9a466439dbb3408008a25283a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5b52c0f5215e43db869233c3d21dc686, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2be66a55356643f59141f8f91ad801ea, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e2a40fcd3deb4d66a678a6ebc487ab29, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/3741a6213a0142509f129c71b3ef1e8e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a2dd2c5b36e94ac6b7b608bd6f6c42f2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13adf8e150c74b85920458acaea5e02d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/1965077d684d4082b064054d4dc2e125, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/bce4698fc8aa4c14bff4a6f3699d3bfe, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a7e6bb3552614e7284b84c095117909a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de3e67ed33084a70baf9e1b92e6632c7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c28d561e7a2e4023b58472ead25dd1b4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7ad7a215de564db9809c62e433b7b398, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/da69bc1a60434f3996679aea86b9611c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9cd5c9aff0164f679370f990b3431f32, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/fe8a1bc2798d42a8bf64010aaf759439, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13a95957b2ac4034bdfe50304316f466, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/ccd1260f509744d8a90b60d2545d88bf, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/6877928b262649c0bc8a750c4cf928f1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3ea72e256284347809a46624b427fdd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/12d9e1c0b50648fb9ff1a080050e1446, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2c91919bbf124473a0d700038222f6a0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7fe857a540e64a62abe4213442c9c0fb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/dad75ada66c3412fb04ca39c00aefc83, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/30dbcba9bcbf46ffa190217ce467f333, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8899de74748847348bb5170f49aef8e7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9f5049cfca4a48929b527d4b3b866f1a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9e80e8f78a4e4981932703852a98b27f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/73e0f81aad4c4ff58cef60d4a44cff99, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8a937e5af1864dbd9e2d09dedefa5256, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e0d160dfa1e94c8483c3a04a6bd4ee05, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/05b2d1ace6f84a64a42be522a22ad206, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/570ed5fbbaec4b2cb66c2dc92873c390, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e29e388e8f9b41678e051ecd19ce829c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/041f0f64ff5e4aa7989f2e62b083f0c1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/924f7fb8a5cc4118ac43a4c02a97ac44, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/f75df679ddc54ef1b4a3ff05d28a9cc1, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de6b40825b094e8e8bfd0b91f7ae4846, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/06c815cba1e8403496990abd27a71651, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/0b8a3b647b1a480a9457f5889a396529, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5863e99bb17e423c8c97ca20cc742366, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/d628762ef3234028aab4b677deb5ae1a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/b8062d0f7ecb4820929f83b203f45a12] to archive 2023-03-26 11:56:33,873 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:56:33,874 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3d526f9a466439dbb3408008a25283a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3d526f9a466439dbb3408008a25283a 2023-03-26 11:56:33,875 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5b52c0f5215e43db869233c3d21dc686 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5b52c0f5215e43db869233c3d21dc686 2023-03-26 11:56:33,876 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2be66a55356643f59141f8f91ad801ea to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2be66a55356643f59141f8f91ad801ea 2023-03-26 11:56:33,877 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e2a40fcd3deb4d66a678a6ebc487ab29 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e2a40fcd3deb4d66a678a6ebc487ab29 2023-03-26 11:56:33,877 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/3741a6213a0142509f129c71b3ef1e8e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/3741a6213a0142509f129c71b3ef1e8e 2023-03-26 11:56:33,878 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a2dd2c5b36e94ac6b7b608bd6f6c42f2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a2dd2c5b36e94ac6b7b608bd6f6c42f2 2023-03-26 11:56:33,879 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13adf8e150c74b85920458acaea5e02d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13adf8e150c74b85920458acaea5e02d 2023-03-26 11:56:33,879 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/1965077d684d4082b064054d4dc2e125 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/1965077d684d4082b064054d4dc2e125 2023-03-26 11:56:33,880 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/bce4698fc8aa4c14bff4a6f3699d3bfe to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/bce4698fc8aa4c14bff4a6f3699d3bfe 2023-03-26 11:56:33,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a7e6bb3552614e7284b84c095117909a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a7e6bb3552614e7284b84c095117909a 2023-03-26 11:56:33,881 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de3e67ed33084a70baf9e1b92e6632c7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de3e67ed33084a70baf9e1b92e6632c7 2023-03-26 11:56:33,882 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c28d561e7a2e4023b58472ead25dd1b4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c28d561e7a2e4023b58472ead25dd1b4 2023-03-26 11:56:33,883 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7ad7a215de564db9809c62e433b7b398 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7ad7a215de564db9809c62e433b7b398 2023-03-26 11:56:33,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/da69bc1a60434f3996679aea86b9611c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/da69bc1a60434f3996679aea86b9611c 2023-03-26 11:56:33,884 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9cd5c9aff0164f679370f990b3431f32 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9cd5c9aff0164f679370f990b3431f32 2023-03-26 11:56:33,885 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/fe8a1bc2798d42a8bf64010aaf759439 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/fe8a1bc2798d42a8bf64010aaf759439 2023-03-26 11:56:33,886 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13a95957b2ac4034bdfe50304316f466 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/13a95957b2ac4034bdfe50304316f466 2023-03-26 11:56:33,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/ccd1260f509744d8a90b60d2545d88bf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/ccd1260f509744d8a90b60d2545d88bf 2023-03-26 11:56:33,887 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/6877928b262649c0bc8a750c4cf928f1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/6877928b262649c0bc8a750c4cf928f1 2023-03-26 11:56:33,888 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3ea72e256284347809a46624b427fdd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/c3ea72e256284347809a46624b427fdd 2023-03-26 11:56:33,889 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/12d9e1c0b50648fb9ff1a080050e1446 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/12d9e1c0b50648fb9ff1a080050e1446 2023-03-26 11:56:33,890 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2c91919bbf124473a0d700038222f6a0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/2c91919bbf124473a0d700038222f6a0 2023-03-26 11:56:33,890 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7fe857a540e64a62abe4213442c9c0fb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/7fe857a540e64a62abe4213442c9c0fb 2023-03-26 11:56:33,891 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/dad75ada66c3412fb04ca39c00aefc83 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/dad75ada66c3412fb04ca39c00aefc83 2023-03-26 11:56:33,892 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/30dbcba9bcbf46ffa190217ce467f333 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/30dbcba9bcbf46ffa190217ce467f333 2023-03-26 11:56:33,893 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8899de74748847348bb5170f49aef8e7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8899de74748847348bb5170f49aef8e7 2023-03-26 11:56:33,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9f5049cfca4a48929b527d4b3b866f1a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9f5049cfca4a48929b527d4b3b866f1a 2023-03-26 11:56:33,894 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9e80e8f78a4e4981932703852a98b27f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/9e80e8f78a4e4981932703852a98b27f 2023-03-26 11:56:33,895 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/73e0f81aad4c4ff58cef60d4a44cff99 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/73e0f81aad4c4ff58cef60d4a44cff99 2023-03-26 11:56:33,896 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8a937e5af1864dbd9e2d09dedefa5256 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/8a937e5af1864dbd9e2d09dedefa5256 2023-03-26 11:56:33,897 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e0d160dfa1e94c8483c3a04a6bd4ee05 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e0d160dfa1e94c8483c3a04a6bd4ee05 2023-03-26 11:56:33,898 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/05b2d1ace6f84a64a42be522a22ad206 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/05b2d1ace6f84a64a42be522a22ad206 2023-03-26 11:56:33,898 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/570ed5fbbaec4b2cb66c2dc92873c390 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/570ed5fbbaec4b2cb66c2dc92873c390 2023-03-26 11:56:33,899 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e29e388e8f9b41678e051ecd19ce829c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/e29e388e8f9b41678e051ecd19ce829c 2023-03-26 11:56:33,900 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/041f0f64ff5e4aa7989f2e62b083f0c1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/041f0f64ff5e4aa7989f2e62b083f0c1 2023-03-26 11:56:33,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/924f7fb8a5cc4118ac43a4c02a97ac44 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/924f7fb8a5cc4118ac43a4c02a97ac44 2023-03-26 11:56:33,901 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/f75df679ddc54ef1b4a3ff05d28a9cc1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/f75df679ddc54ef1b4a3ff05d28a9cc1 2023-03-26 11:56:33,902 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de6b40825b094e8e8bfd0b91f7ae4846 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/de6b40825b094e8e8bfd0b91f7ae4846 2023-03-26 11:56:33,903 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/06c815cba1e8403496990abd27a71651 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/06c815cba1e8403496990abd27a71651 2023-03-26 11:56:33,904 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/0b8a3b647b1a480a9457f5889a396529 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/0b8a3b647b1a480a9457f5889a396529 2023-03-26 11:56:33,905 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5863e99bb17e423c8c97ca20cc742366 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/5863e99bb17e423c8c97ca20cc742366 2023-03-26 11:56:33,906 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/d628762ef3234028aab4b677deb5ae1a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/d628762ef3234028aab4b677deb5ae1a 2023-03-26 11:56:33,907 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/b8062d0f7ecb4820929f83b203f45a12 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/b8062d0f7ecb4820929f83b203f45a12 2023-03-26 11:56:33,910 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fcfd46e696bc4e44890cc25cd51e8ee8, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/e88f54432e674fdfb523eb5d26135ceb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/39238d8380664723b1681c735d072c01, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/4c973dfc185d4d2e8ecf0ae7e6222893, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/5b0e139a2dfb4fd39f4c432ef6c0443b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/919bf69243124aa7a7bb80d426b355c4, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a896df2ade074958b64fac5762d6dbc2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/51731882efee419488dbaee555edac0a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/b5b430022a4546f784fc2669a3e11f2c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/af63d69742b34ffb83050a150f73da4c, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d4da6d34945348b78e397990746881e6, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/49139d2f6c984b219f922d19af20e93d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd372eaa11b64a259c86e40c7548e269, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/dd67ee546a6b419a8fc6a5f7e0505e0a, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c8ebffafb3ad45908b67c1b288ec8f45, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68dac10c824b4737b6ea73eaa7d2f567, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d611bb0d41144fa4a3b8f507a2945082, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68ff5dc54ef14f7bbf4060b67f1ac9a9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/808d5d74ae614ec2a54c4f3766338ec7, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a65101d67253470fadecf451c75092fb, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d9b831d328e54cf784913467e6c6e215, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/7c5b61063652463692c0e12b93202a84, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/1ae46615dec74091a71b359784de9eb2, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a21b90734b654adcbb452fe9caaaf574, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/11a0587e2ef44c29aebb79ecca4349e0, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/ffdf9502eefe49e4a3074c8e1e28b434, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9184ee8dc8ef43dca8fd5f9f3adee32b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8526df90c0154f0ebe3a247bdc49a37f, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3d88f71853da400594d89d304878ff2d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8861e1bbe347413c8b47252a41f15a2e, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/219712a3fca745769910643259be0b22, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/6d72fc7536d84ff1b6b43c3e7b54dfb9, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c90b4da55e024e70a10681db77c74f28, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fa8604659a624cfeb6ff636eb9433a01, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fc492764d13843fa915b8a57757a4157, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3070c7e101fe4933b4a9fa88b5fa1c9b, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c11cee48fe63460f976080fc23efc441, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd18be34d9ad44bb8eddcf9e4f2b509d, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bbfcac663d7e4f2a8c200a87c2c6d5ec, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f5624f8b58a742e3a9c11c6e82429ccd, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c93aeb9c53a940c5a2a566a5664093a8] to archive 2023-03-26 11:56:33,911 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-03-26 11:56:33,912 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fcfd46e696bc4e44890cc25cd51e8ee8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fcfd46e696bc4e44890cc25cd51e8ee8 2023-03-26 11:56:33,913 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/e88f54432e674fdfb523eb5d26135ceb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/e88f54432e674fdfb523eb5d26135ceb 2023-03-26 11:56:33,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/39238d8380664723b1681c735d072c01 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/39238d8380664723b1681c735d072c01 2023-03-26 11:56:33,914 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/4c973dfc185d4d2e8ecf0ae7e6222893 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/4c973dfc185d4d2e8ecf0ae7e6222893 2023-03-26 11:56:33,915 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/5b0e139a2dfb4fd39f4c432ef6c0443b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/5b0e139a2dfb4fd39f4c432ef6c0443b 2023-03-26 11:56:33,916 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/919bf69243124aa7a7bb80d426b355c4 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/919bf69243124aa7a7bb80d426b355c4 2023-03-26 11:56:33,917 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a896df2ade074958b64fac5762d6dbc2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a896df2ade074958b64fac5762d6dbc2 2023-03-26 11:56:33,918 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/51731882efee419488dbaee555edac0a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/51731882efee419488dbaee555edac0a 2023-03-26 11:56:33,919 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/b5b430022a4546f784fc2669a3e11f2c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/b5b430022a4546f784fc2669a3e11f2c 2023-03-26 11:56:33,920 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/af63d69742b34ffb83050a150f73da4c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/af63d69742b34ffb83050a150f73da4c 2023-03-26 11:56:33,921 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d4da6d34945348b78e397990746881e6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d4da6d34945348b78e397990746881e6 2023-03-26 11:56:33,922 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/49139d2f6c984b219f922d19af20e93d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/49139d2f6c984b219f922d19af20e93d 2023-03-26 11:56:33,923 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd372eaa11b64a259c86e40c7548e269 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd372eaa11b64a259c86e40c7548e269 2023-03-26 11:56:33,924 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/dd67ee546a6b419a8fc6a5f7e0505e0a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/dd67ee546a6b419a8fc6a5f7e0505e0a 2023-03-26 11:56:33,925 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c8ebffafb3ad45908b67c1b288ec8f45 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c8ebffafb3ad45908b67c1b288ec8f45 2023-03-26 11:56:33,926 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68dac10c824b4737b6ea73eaa7d2f567 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68dac10c824b4737b6ea73eaa7d2f567 2023-03-26 11:56:33,927 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d611bb0d41144fa4a3b8f507a2945082 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d611bb0d41144fa4a3b8f507a2945082 2023-03-26 11:56:33,927 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68ff5dc54ef14f7bbf4060b67f1ac9a9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/68ff5dc54ef14f7bbf4060b67f1ac9a9 2023-03-26 11:56:33,928 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/808d5d74ae614ec2a54c4f3766338ec7 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/808d5d74ae614ec2a54c4f3766338ec7 2023-03-26 11:56:33,929 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a65101d67253470fadecf451c75092fb to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a65101d67253470fadecf451c75092fb 2023-03-26 11:56:33,930 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d9b831d328e54cf784913467e6c6e215 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/d9b831d328e54cf784913467e6c6e215 2023-03-26 11:56:33,930 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/7c5b61063652463692c0e12b93202a84 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/7c5b61063652463692c0e12b93202a84 2023-03-26 11:56:33,931 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/1ae46615dec74091a71b359784de9eb2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/1ae46615dec74091a71b359784de9eb2 2023-03-26 11:56:33,932 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a21b90734b654adcbb452fe9caaaf574 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/a21b90734b654adcbb452fe9caaaf574 2023-03-26 11:56:33,933 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/11a0587e2ef44c29aebb79ecca4349e0 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/11a0587e2ef44c29aebb79ecca4349e0 2023-03-26 11:56:33,934 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/ffdf9502eefe49e4a3074c8e1e28b434 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/ffdf9502eefe49e4a3074c8e1e28b434 2023-03-26 11:56:33,935 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9184ee8dc8ef43dca8fd5f9f3adee32b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9184ee8dc8ef43dca8fd5f9f3adee32b 2023-03-26 11:56:33,936 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8526df90c0154f0ebe3a247bdc49a37f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8526df90c0154f0ebe3a247bdc49a37f 2023-03-26 11:56:33,937 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3d88f71853da400594d89d304878ff2d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3d88f71853da400594d89d304878ff2d 2023-03-26 11:56:33,937 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8861e1bbe347413c8b47252a41f15a2e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/8861e1bbe347413c8b47252a41f15a2e 2023-03-26 11:56:33,938 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/219712a3fca745769910643259be0b22 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/219712a3fca745769910643259be0b22 2023-03-26 11:56:33,939 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/6d72fc7536d84ff1b6b43c3e7b54dfb9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/6d72fc7536d84ff1b6b43c3e7b54dfb9 2023-03-26 11:56:33,939 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c90b4da55e024e70a10681db77c74f28 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c90b4da55e024e70a10681db77c74f28 2023-03-26 11:56:33,940 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fa8604659a624cfeb6ff636eb9433a01 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fa8604659a624cfeb6ff636eb9433a01 2023-03-26 11:56:33,941 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fc492764d13843fa915b8a57757a4157 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/fc492764d13843fa915b8a57757a4157 2023-03-26 11:56:33,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3070c7e101fe4933b4a9fa88b5fa1c9b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/3070c7e101fe4933b4a9fa88b5fa1c9b 2023-03-26 11:56:33,942 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c11cee48fe63460f976080fc23efc441 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c11cee48fe63460f976080fc23efc441 2023-03-26 11:56:33,943 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd18be34d9ad44bb8eddcf9e4f2b509d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bd18be34d9ad44bb8eddcf9e4f2b509d 2023-03-26 11:56:33,944 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bbfcac663d7e4f2a8c200a87c2c6d5ec to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/bbfcac663d7e4f2a8c200a87c2c6d5ec 2023-03-26 11:56:33,944 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f5624f8b58a742e3a9c11c6e82429ccd to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f5624f8b58a742e3a9c11c6e82429ccd 2023-03-26 11:56:33,945 DEBUG [StoreCloser-TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c93aeb9c53a940c5a2a566a5664093a8 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/c93aeb9c53a940c5a2a566a5664093a8 2023-03-26 11:56:33,948 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/recovered.edits/1037.seqid, newMaxSeqId=1037, maxSeqId=4 2023-03-26 11:56:33,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50. 2023-03-26 11:56:33,950 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 5890b64a473c744d0cc3b74eead11b50: 2023-03-26 11:56:33,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:33,951 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=5890b64a473c744d0cc3b74eead11b50, regionState=CLOSED 2023-03-26 11:56:33,951 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1679831793951"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1679831793951"}]},"ts":"1679831793951"} 2023-03-26 11:56:33,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-03-26 11:56:33,953 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 5890b64a473c744d0cc3b74eead11b50, server=jenkins-hbase4.apache.org,38971,1679831615531 in 1.5420 sec 2023-03-26 11:56:33,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-03-26 11:56:33,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=5890b64a473c744d0cc3b74eead11b50, UNASSIGN in 1.5460 sec 2023-03-26 11:56:33,954 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1679831793954"}]},"ts":"1679831793954"} 2023-03-26 11:56:33,955 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-03-26 11:56:33,958 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-03-26 11:56:33,959 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5570 sec 2023-03-26 11:56:34,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-03-26 11:56:34,508 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-03-26 11:56:34,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-03-26 11:56:34,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:34,511 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:34,512 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:34,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-26 11:56:34,515 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:34,517 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C, FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/recovered.edits] 2023-03-26 11:56:34,520 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4cf6e028123d47a886594d57f361bd49 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/4cf6e028123d47a886594d57f361bd49 2023-03-26 11:56:34,521 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8c197e4601d84096b3090ade67bfe97c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/8c197e4601d84096b3090ade67bfe97c 2023-03-26 11:56:34,521 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ad994567153a433c8dfbd88a401d7525 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/A/ad994567153a433c8dfbd88a401d7525 2023-03-26 11:56:34,523 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/22c523b8f1eb455ea860074fedd2e9cf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/22c523b8f1eb455ea860074fedd2e9cf 2023-03-26 11:56:34,524 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/616afd87e6304238845d6e5c55986880 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/616afd87e6304238845d6e5c55986880 2023-03-26 11:56:34,525 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a284538616d9475b94c971a83fe57a2b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/B/a284538616d9475b94c971a83fe57a2b 2023-03-26 11:56:34,527 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/58d03033a3194eee978ee3ca171d406f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/58d03033a3194eee978ee3ca171d406f 2023-03-26 11:56:34,527 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9a028c87b7d94e1c924bf9353aa380de to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/9a028c87b7d94e1c924bf9353aa380de 2023-03-26 11:56:34,528 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f0ae9b27c71e4b6daef09dc8ccfeab5f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/C/f0ae9b27c71e4b6daef09dc8ccfeab5f 2023-03-26 11:56:34,530 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/recovered.edits/1037.seqid to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50/recovered.edits/1037.seqid 2023-03-26 11:56:34,530 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/.tmp/data/default/TestAcidGuarantees/5890b64a473c744d0cc3b74eead11b50 2023-03-26 11:56:34,530 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-03-26 11:56:34,531 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 11:56:34,531 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-03-26 11:56:34,533 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032606aa51b714af4d91ae89ce2aeb1f1867 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032606aa51b714af4d91ae89ce2aeb1f1867 2023-03-26 11:56:34,534 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261e6d201759384238950c86714411859e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303261e6d201759384238950c86714411859e 2023-03-26 11:56:34,535 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326295680c782184dcb90581dcddf61f62d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326295680c782184dcb90581dcddf61f62d 2023-03-26 11:56:34,536 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326306de39e66174d0994a3688a5373ebdf to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326306de39e66174d0994a3688a5373ebdf 2023-03-26 11:56:34,537 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263645417c2890419baaca01fce66fa347 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263645417c2890419baaca01fce66fa347 2023-03-26 11:56:34,538 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263b58ae5fa02b46fbb42e7b68c1e702fc to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303263b58ae5fa02b46fbb42e7b68c1e702fc 2023-03-26 11:56:34,539 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264ad61d2ffe344a25bc57f241d3a7b611 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303264ad61d2ffe344a25bc57f241d3a7b611 2023-03-26 11:56:34,540 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032653a2519c2e69441287fb307f25e0b987 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032653a2519c2e69441287fb307f25e0b987 2023-03-26 11:56:34,541 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303266b972cb3e4b04bacb708392483985bd9 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303266b972cb3e4b04bacb708392483985bd9 2023-03-26 11:56:34,542 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267b5dabb2feda456297142e9117f7e1ff to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267b5dabb2feda456297142e9117f7e1ff 2023-03-26 11:56:34,542 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267ea03a8a551f4bd49ee8fae3c24d624e to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267ea03a8a551f4bd49ee8fae3c24d624e 2023-03-26 11:56:34,543 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267f0e10315727486e9b4a6ab5f218000a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303267f0e10315727486e9b4a6ab5f218000a 2023-03-26 11:56:34,544 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682af3dd08c744eb489bdb9c9b9f2801b to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023032682af3dd08c744eb489bdb9c9b9f2801b 2023-03-26 11:56:34,545 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269091a5bbe1c740cbb3c258177aac5599 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269091a5bbe1c740cbb3c258177aac5599 2023-03-26 11:56:34,546 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326932768420e344784a3a7cc8b3cc007a2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326932768420e344784a3a7cc8b3cc007a2 2023-03-26 11:56:34,547 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326996cce569dcd4a5db9f0036f6353c1f2 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326996cce569dcd4a5db9f0036f6353c1f2 2023-03-26 11:56:34,548 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269caca11d1df44beb982c870a5768ded1 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202303269caca11d1df44beb982c870a5768ded1 2023-03-26 11:56:34,549 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a0110ef7141e47d28ff1a89f3922356f to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a0110ef7141e47d28ff1a89f3922356f 2023-03-26 11:56:34,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a073a524f4a546c1948c08f8349eec5c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a073a524f4a546c1948c08f8349eec5c 2023-03-26 11:56:34,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a65ed4bd1a4440d0ac35db33ebe74f86 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a65ed4bd1a4440d0ac35db33ebe74f86 2023-03-26 11:56:34,551 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a8b1711494b64f9e94ef06fedde93f0a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a8b1711494b64f9e94ef06fedde93f0a 2023-03-26 11:56:34,552 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a985c8f2beda4e3289d1b23dfe7608ca to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a985c8f2beda4e3289d1b23dfe7608ca 2023-03-26 11:56:34,553 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a9af9e2540fe420e9c9eeae5596fc190 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326a9af9e2540fe420e9c9eeae5596fc190 2023-03-26 11:56:34,554 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326aa884b10e30246debf75f1a8783dbc1a to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326aa884b10e30246debf75f1a8783dbc1a 2023-03-26 11:56:34,555 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b5a903f0eef24c3da0e65925ce060c31 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b5a903f0eef24c3da0e65925ce060c31 2023-03-26 11:56:34,556 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b6e0722e7743428da1f1125e0bbc26e6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b6e0722e7743428da1f1125e0bbc26e6 2023-03-26 11:56:34,557 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b8bcc14a87a44012a46046fe6381bee6 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326b8bcc14a87a44012a46046fe6381bee6 2023-03-26 11:56:34,557 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c67167a58d074d95b549c1ef92431eed to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326c67167a58d074d95b549c1ef92431eed 2023-03-26 11:56:34,558 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326cdb9cfe0591b4c4f999d3ffcfe1dfbec to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326cdb9cfe0591b4c4f999d3ffcfe1dfbec 2023-03-26 11:56:34,559 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ddb172ddf6fd466c8c2e9cca9e978210 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326ddb172ddf6fd466c8c2e9cca9e978210 2023-03-26 11:56:34,560 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e0dcb323ddc14e4caec6b6959eb43045 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e0dcb323ddc14e4caec6b6959eb43045 2023-03-26 11:56:34,561 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e6701e156cef4e76bc6bc6fcb3b93763 to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326e6701e156cef4e76bc6bc6fcb3b93763 2023-03-26 11:56:34,562 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326edf3f0ec7bd64b5e97fd03e46feced6c to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326edf3f0ec7bd64b5e97fd03e46feced6c 2023-03-26 11:56:34,563 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f6d286b9f3584942a8092ab82fd8d18d to hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230326f6d286b9f3584942a8092ab82fd8d18d 2023-03-26 11:56:34,563 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-03-26 11:56:34,565 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:34,566 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-03-26 11:56:34,568 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-03-26 11:56:34,568 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:34,568 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-03-26 11:56:34,568 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1679831794568"}]},"ts":"9223372036854775807"} 2023-03-26 11:56:34,570 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-03-26 11:56:34,570 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 5890b64a473c744d0cc3b74eead11b50, NAME => 'TestAcidGuarantees,,1679831760559.5890b64a473c744d0cc3b74eead11b50.', STARTKEY => '', ENDKEY => ''}] 2023-03-26 11:56:34,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-03-26 11:56:34,570 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1679831794570"}]},"ts":"9223372036854775807"} 2023-03-26 11:56:34,571 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-03-26 11:56:34,573 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-03-26 11:56:34,573 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 64 msec 2023-03-26 11:56:34,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=43599] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-03-26 11:56:34,613 INFO [Listener at localhost/46409] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-03-26 11:56:34,627 INFO [Listener at localhost/46409] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=225 (was 225), OpenFileDescriptor=510 (was 554), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=371 (was 361) - SystemLoadAverage LEAK? -, ProcessCount=168 (was 169), AvailableMemoryMB=9042 (was 9440) 2023-03-26 11:56:34,628 INFO [Listener at localhost/46409] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-03-26 11:56:34,628 INFO [Listener at localhost/46409] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-03-26 11:56:34,628 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3ddd91d3 to 127.0.0.1:53955 2023-03-26 11:56:34,628 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:34,629 DEBUG [Listener at localhost/46409] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-03-26 11:56:34,629 DEBUG [Listener at localhost/46409] util.JVMClusterUtil(257): Found active master hash=1341213680, stopped=false 2023-03-26 11:56:34,629 INFO [Listener at localhost/46409] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:56:34,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-26 11:56:34,630 INFO [Listener at localhost/46409] procedure2.ProcedureExecutor(629): Stopping 2023-03-26 11:56:34,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:56:34,630 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-03-26 11:56:34,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 11:56:34,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-03-26 11:56:34,631 DEBUG [Listener at localhost/46409] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d3e3c2 to 127.0.0.1:53955 2023-03-26 11:56:34,631 DEBUG [Listener at localhost/46409] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:34,631 INFO [Listener at localhost/46409] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,38971,1679831615531' ***** 2023-03-26 11:56:34,631 INFO [Listener at localhost/46409] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-03-26 11:56:34,631 INFO [RS:0;jenkins-hbase4:38971] regionserver.HeapMemoryManager(220): Stopping 2023-03-26 11:56:34,631 INFO [RS:0;jenkins-hbase4:38971] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-03-26 11:56:34,632 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-03-26 11:56:34,632 INFO [RS:0;jenkins-hbase4:38971] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-03-26 11:56:34,632 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(3304): Received CLOSE for fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:56:34,632 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:34,632 DEBUG [RS:0;jenkins-hbase4:38971] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23e83673 to 127.0.0.1:53955 2023-03-26 11:56:34,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing fc411bea9b9107578035998a3aeb7bf5, disabling compactions & flushes 2023-03-26 11:56:34,633 DEBUG [RS:0;jenkins-hbase4:38971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:34,633 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:56:34,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:56:34,633 INFO [RS:0;jenkins-hbase4:38971] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-03-26 11:56:34,633 INFO [RS:0;jenkins-hbase4:38971] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-03-26 11:56:34,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. after waiting 0 ms 2023-03-26 11:56:34,633 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:56:34,633 INFO [RS:0;jenkins-hbase4:38971] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-03-26 11:56:34,634 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing fc411bea9b9107578035998a3aeb7bf5 1/1 column families, dataSize=78 B heapSize=488 B 2023-03-26 11:56:34,634 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-03-26 11:56:34,634 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-03-26 11:56:34,634 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, fc411bea9b9107578035998a3aeb7bf5=hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5.} 2023-03-26 11:56:34,634 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-03-26 11:56:34,634 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-03-26 11:56:34,634 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-03-26 11:56:34,635 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 1 ms 2023-03-26 11:56:34,635 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-03-26 11:56:34,635 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-03-26 11:56:34,636 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1505): Waiting on 1588230740, fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:56:34,642 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/.tmp/info/a6515bb821ea4e978b1309859a01cff2 2023-03-26 11:56:34,646 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a6515bb821ea4e978b1309859a01cff2 2023-03-26 11:56:34,680 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-03-26 11:56:34,836 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1505): Waiting on 1588230740, fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:56:34,931 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-03-26 11:56:34,931 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-03-26 11:56:35,037 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1505): Waiting on 1588230740, fc411bea9b9107578035998a3aeb7bf5 2023-03-26 11:56:35,042 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/.tmp/info/c83ae2b4901e411894ae949117568ad7 2023-03-26 11:56:35,046 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/.tmp/info/c83ae2b4901e411894ae949117568ad7 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/info/c83ae2b4901e411894ae949117568ad7 2023-03-26 11:56:35,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/info/c83ae2b4901e411894ae949117568ad7, entries=2, sequenceid=6, filesize=4.8 K 2023-03-26 11:56:35,050 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for fc411bea9b9107578035998a3aeb7bf5 in 416ms, sequenceid=6, compaction requested=false 2023-03-26 11:56:35,055 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/namespace/fc411bea9b9107578035998a3aeb7bf5/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-03-26 11:56:35,056 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:56:35,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for fc411bea9b9107578035998a3aeb7bf5: 2023-03-26 11:56:35,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1679831617478.fc411bea9b9107578035998a3aeb7bf5. 2023-03-26 11:56:35,056 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/.tmp/rep_barrier/34721501f7024083b29c3b94991d5851 2023-03-26 11:56:35,059 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 34721501f7024083b29c3b94991d5851 2023-03-26 11:56:35,237 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-26 11:56:35,437 DEBUG [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-03-26 11:56:35,465 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/.tmp/table/4a1665e7e6ab49028265f31ed63a3c8e 2023-03-26 11:56:35,467 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a1665e7e6ab49028265f31ed63a3c8e 2023-03-26 11:56:35,468 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/.tmp/info/a6515bb821ea4e978b1309859a01cff2 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/info/a6515bb821ea4e978b1309859a01cff2 2023-03-26 11:56:35,471 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for a6515bb821ea4e978b1309859a01cff2 2023-03-26 11:56:35,471 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/info/a6515bb821ea4e978b1309859a01cff2, entries=22, sequenceid=93, filesize=7.1 K 2023-03-26 11:56:35,472 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/.tmp/rep_barrier/34721501f7024083b29c3b94991d5851 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/rep_barrier/34721501f7024083b29c3b94991d5851 2023-03-26 11:56:35,475 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 34721501f7024083b29c3b94991d5851 2023-03-26 11:56:35,475 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/rep_barrier/34721501f7024083b29c3b94991d5851, entries=6, sequenceid=93, filesize=5.5 K 2023-03-26 11:56:35,475 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/.tmp/table/4a1665e7e6ab49028265f31ed63a3c8e as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/table/4a1665e7e6ab49028265f31ed63a3c8e 2023-03-26 11:56:35,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 4a1665e7e6ab49028265f31ed63a3c8e 2023-03-26 11:56:35,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/table/4a1665e7e6ab49028265f31ed63a3c8e, entries=9, sequenceid=93, filesize=5.5 K 2023-03-26 11:56:35,478 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 843ms, sequenceid=93, compaction requested=false 2023-03-26 11:56:35,486 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-03-26 11:56:35,486 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-03-26 11:56:35,487 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-03-26 11:56:35,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-03-26 11:56:35,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-03-26 11:56:35,637 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,38971,1679831615531; all regions closed. 2023-03-26 11:56:35,645 DEBUG [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/oldWALs 2023-03-26 11:56:35,645 INFO [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38971%2C1679831615531.meta:.meta(num 1679831617238) 2023-03-26 11:56:35,649 DEBUG [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/oldWALs 2023-03-26 11:56:35,650 INFO [RS:0;jenkins-hbase4:38971] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C38971%2C1679831615531:(num 1679831617024) 2023-03-26 11:56:35,650 DEBUG [RS:0;jenkins-hbase4:38971] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:35,650 INFO [RS:0;jenkins-hbase4:38971] regionserver.LeaseManager(133): Closed leases 2023-03-26 11:56:35,650 INFO [RS:0;jenkins-hbase4:38971] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS, ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS] on shutdown 2023-03-26 11:56:35,650 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-26 11:56:35,652 INFO [RS:0;jenkins-hbase4:38971] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:38971 2023-03-26 11:56:35,657 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,38971,1679831615531 2023-03-26 11:56:35,657 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-26 11:56:35,657 ERROR [Listener at localhost/46409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@2e16bb11 rejected from java.util.concurrent.ThreadPoolExecutor@2a292a0b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 11:56:35,657 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-03-26 11:56:35,657 ERROR [Listener at localhost/46409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@455bef2d rejected from java.util.concurrent.ThreadPoolExecutor@2a292a0b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 11:56:35,658 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,38971,1679831615531] 2023-03-26 11:56:35,659 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,38971,1679831615531; numProcessing=1 2023-03-26 11:56:35,660 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,38971,1679831615531 already deleted, retry=false 2023-03-26 11:56:35,660 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,38971,1679831615531 expired; onlineServers=0 2023-03-26 11:56:35,660 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,43599,1679831614279' ***** 2023-03-26 11:56:35,660 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-03-26 11:56:35,660 DEBUG [M:0;jenkins-hbase4:43599] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@ecc41cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-03-26 11:56:35,660 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,43599,1679831614279 2023-03-26 11:56:35,660 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,43599,1679831614279; all regions closed. 2023-03-26 11:56:35,660 DEBUG [M:0;jenkins-hbase4:43599] ipc.AbstractRpcClient(494): Stopping rpc client 2023-03-26 11:56:35,660 DEBUG [M:0;jenkins-hbase4:43599] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-03-26 11:56:35,660 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-03-26 11:56:35,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679831616736] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1679831616736,5,FailOnTimeoutGroup] 2023-03-26 11:56:35,660 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679831616736] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1679831616736,5,FailOnTimeoutGroup] 2023-03-26 11:56:35,660 DEBUG [M:0;jenkins-hbase4:43599] cleaner.HFileCleaner(317): Stopping file delete threads 2023-03-26 11:56:35,661 INFO [M:0;jenkins-hbase4:43599] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-03-26 11:56:35,661 INFO [M:0;jenkins-hbase4:43599] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-03-26 11:56:35,661 INFO [M:0;jenkins-hbase4:43599] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-03-26 11:56:35,661 DEBUG [M:0;jenkins-hbase4:43599] master.HMaster(1512): Stopping service threads 2023-03-26 11:56:35,662 INFO [M:0;jenkins-hbase4:43599] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-03-26 11:56:35,662 ERROR [M:0;jenkins-hbase4:43599] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-03-26 11:56:35,662 INFO [M:0;jenkins-hbase4:43599] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-03-26 11:56:35,662 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-03-26 11:56:35,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-03-26 11:56:35,663 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-03-26 11:56:35,663 DEBUG [M:0;jenkins-hbase4:43599] zookeeper.ZKUtil(398): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-03-26 11:56:35,663 WARN [M:0;jenkins-hbase4:43599] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-03-26 11:56:35,663 INFO [M:0;jenkins-hbase4:43599] assignment.AssignmentManager(315): Stopping assignment manager 2023-03-26 11:56:35,663 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-03-26 11:56:35,663 INFO [M:0;jenkins-hbase4:43599] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-03-26 11:56:35,664 DEBUG [M:0;jenkins-hbase4:43599] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-03-26 11:56:35,664 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:56:35,664 DEBUG [M:0;jenkins-hbase4:43599] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:56:35,664 DEBUG [M:0;jenkins-hbase4:43599] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-03-26 11:56:35,664 DEBUG [M:0;jenkins-hbase4:43599] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:56:35,664 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=345.07 KB heapSize=393.73 KB 2023-03-26 11:56:35,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 11:56:35,759 INFO [RS:0;jenkins-hbase4:38971] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,38971,1679831615531; zookeeper connection closed. 2023-03-26 11:56:35,759 ERROR [Listener at localhost/46409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3d141a0c rejected from java.util.concurrent.ThreadPoolExecutor@2a292a0b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 11:56:35,759 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): regionserver:38971-0x10120ac29a00001, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 11:56:35,759 ERROR [Listener at localhost/46409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@15bcea2f rejected from java.util.concurrent.ThreadPoolExecutor@2a292a0b[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2389] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 11:56:35,759 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@415994fc] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@415994fc 2023-03-26 11:56:35,760 INFO [Listener at localhost/46409] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-03-26 11:56:36,078 INFO [M:0;jenkins-hbase4:43599] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=345.07 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/97e1e5045b2642b39b5e6c8526305227 2023-03-26 11:56:36,082 INFO [M:0;jenkins-hbase4:43599] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 97e1e5045b2642b39b5e6c8526305227 2023-03-26 11:56:36,083 DEBUG [M:0;jenkins-hbase4:43599] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/97e1e5045b2642b39b5e6c8526305227 as hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/97e1e5045b2642b39b5e6c8526305227 2023-03-26 11:56:36,086 INFO [M:0;jenkins-hbase4:43599] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 97e1e5045b2642b39b5e6c8526305227 2023-03-26 11:56:36,086 INFO [M:0;jenkins-hbase4:43599] regionserver.HStore(1080): Added hdfs://localhost:46069/user/jenkins/test-data/63a3a84e-d807-0f1e-b7ae-7e1247b150f1/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/97e1e5045b2642b39b5e6c8526305227, entries=65, sequenceid=578, filesize=24.7 K 2023-03-26 11:56:36,086 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegion(2948): Finished flush of dataSize ~345.07 KB/353349, heapSize ~393.72 KB/403168, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 422ms, sequenceid=578, compaction requested=false 2023-03-26 11:56:36,088 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-03-26 11:56:36,088 DEBUG [M:0;jenkins-hbase4:43599] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-03-26 11:56:36,091 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-03-26 11:56:36,091 INFO [M:0;jenkins-hbase4:43599] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-03-26 11:56:36,091 INFO [M:0;jenkins-hbase4:43599] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:43599 2023-03-26 11:56:36,093 DEBUG [M:0;jenkins-hbase4:43599] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,43599,1679831614279 already deleted, retry=false 2023-03-26 11:56:36,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 11:56:36,194 INFO [M:0;jenkins-hbase4:43599] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,43599,1679831614279; zookeeper connection closed. 2023-03-26 11:56:36,194 ERROR [Listener at localhost/46409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@1df3aaf7 rejected from java.util.concurrent.ThreadPoolExecutor@4b88d3ca[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3995] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 11:56:36,194 DEBUG [Listener at localhost/46409-EventThread] zookeeper.ZKWatcher(600): master:43599-0x10120ac29a00000, quorum=127.0.0.1:53955, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-03-26 11:56:36,195 ERROR [Listener at localhost/46409-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d3319d1 rejected from java.util.concurrent.ThreadPoolExecutor@4b88d3ca[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 3995] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-03-26 11:56:36,196 WARN [Listener at localhost/46409] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-03-26 11:56:36,200 INFO [Listener at localhost/46409] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-26 11:56:36,247 WARN [BP-1384640899-172.31.14.131-1679831610899 heartbeating to localhost/127.0.0.1:46069] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1384640899-172.31.14.131-1679831610899 (Datanode Uuid cd68a317-7bd1-46b9-b40b-44ceb88f195a) service to localhost/127.0.0.1:46069 2023-03-26 11:56:36,252 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/cluster_2110af56-5f7e-313d-fc62-08bf66541fb8/dfs/data/data1/current/BP-1384640899-172.31.14.131-1679831610899] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-26 11:56:36,253 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/e1014ac2-66d8-6758-1efd-b825e642648d/cluster_2110af56-5f7e-313d-fc62-08bf66541fb8/dfs/data/data2/current/BP-1384640899-172.31.14.131-1679831610899] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-03-26 11:56:36,341 INFO [Listener at localhost/46409] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-03-26 11:56:36,510 INFO [Listener at localhost/46409] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-03-26 11:56:36,696 INFO [Listener at localhost/46409] hbase.HBaseTestingUtility(1293): Minicluster is down